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INTRODUCTION 


The  increasing  costs  of  training  and  the  inherent 
risks  of  flight  in  the  Nap-of-the- Earth  (NOE)  regime  require 
that  the  Army  develop  more  efficient  aircrew  training  programs. 
Previous  research  by  PMA  (Connelly,  Comeau,  Bynum  &  Holman, 
1979)  to  improve  NOE  navigation  evaluation  methods  has  shown 
that  subsequent  improvements  in  the  efficiency  of  NOE  aircrew 
training  programs  will  be  dependent  on  a  research  need  for 
better  quality  data  in  the  evaluation  of  aircrew  proficiency. 

The  quality  of  existing  data  is  constrained  by  the  process  of 
instructor  pilot  observation  as  its  source.  The  instructor 
pilot  has  other  duties,  such  as  safecy  and  backup  pilot,  which 
must  take  priority  over  any  formal  observer  or  data  recording 
functions  needed  to  support  research  analyses.  Therefore,  * 
data  yielded  by  instructor  pilot  observations  does  not  readily 
allow  for  detailed  investigations  of  such  factors  as  aircrew 
workload  or  specific  NOE  mission  requirements  and  their 
effects  on  instructor  pilot  ratings  of  overall  quality  of  aircrew 
performance . 


The  US  Army  Aviation  Center  is  studying  the 
development  and  installation  of  a  fully  instrumented  NOE  training 
and  test  range.  This  range  will  have  electronic  technology  to 
collect,  via  telemetry,  comprehensive  time  sequenced  data 
on  aircraft  location,  aircraft  attitude,  and  aircraft  control 
states  at  all  times  during  selected  flight  sorties  within  the  range 
boundaries.  The  time  sequenced  observations  provided  by  this 
instrumented  range  will  yield  an  immediate  solution  to  the  re¬ 
search  need  for  greater  levels  of  detail  in  performance  obser¬ 
vation.  However,  a  concomitant  result  wi<l  be  a  massive  data 
base  for  each  flight  sortie. 

Specialized  methods  of  analysis  must  be  developed 
to  tally  utilize  the  resulting  level  of  detail  because  existing 
statistical  models  give  equal  weight  to  each  data  point 
in  this  massive  data  base.  Equal  weighting  of  each  data  point 
is  known  to  be  invalid  because  previous  research  by  PMA  (Connelly  & 
Loental ,  1974a)  has  shoim  that  the  probability  of  aircrew  perfor¬ 
mance  errors  will  depend  on  particular  locations,  events,  or 
functional  segments  within  the  mission  flight  profile. 
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A  detailed  analysis  of  performance  of  various  types 
of  aircraft  missions  has  shown  that  the  significance  of  flight  errors 
and  flight  control  styles  to  mission  success  is  not  uniform 
over  the  total  mission  profile.  For  example,  rapid  reduction 
of  a  flight  error  such  as  an  altitude  error  may  bo  important 
to  mission  success  at  one  point  in  the  mission,  out  is  of 
less  importance,  and  perhaps  even  '.wasteful  of  crew  memoers* 
energy,  at  other  points  in  the  mission. 

Evidence  of  the  varying  importance  of  control  is 
obtained  from  many  sources.  In  an  analysis  of  an  F-106  attack 
mission,  Connelly  &  Loental  (1974a)  showed  that  superior  per¬ 
formers  used  different  control  policies  for  each  of  the  three 
strategies  of  the  mission  (Prelock-on,  Lock  on.  Postlock-on) . 

Pilots  who  did  not  perform  as  well  on  the  total  mission  did  not 
use  that  varying  control  strategy.  In  another  example,  Connelly, 
Schuler,  Bourne, &  Knoop  (1971),  in  developing  performance 
measures  for  undergraduate  students  flying  pilot  contact 
maneuvers,  ound  that  specified  relationships  between  pitch 
and  roll  were  not  rated  equally  by  instructors  at  all  points 
in  the  maneuver.  Coordination  appeared  to  be  most  important 
at  the  beginning  and  the  end  of  the  maneuver.  In  yet  another 
illustration  of  this  point,  Connelly,  et  ai,  (1979)  found  that 
performance  of  NOE  navigators  was  sensitive  to  combinations 
of  terrain  and  flight  path  conditions  along  the  specified  route. 

In  some  terrain  conditions  the  probability  of  success  was  found 
to  be  high  while  other  terrain  conditions  led  to  more  frequent 
navigation  errors. 

In  the  past,  the  measurement  of  mission  performance 
was  limited  to  summary  measures  which  provided  a  single 
score  for  the  total  mission  performance.  As  a  result,  cnese 
measures  could  not  be  used  to  evaluate  the  varying  sensitivity 
of  control  during  the  mission.  More  recently,  because 
sensitive  measures  have  been  required,  the  technology  for 
these  measures  has  been  developed.  In  addition,  the  use  of 
automatic  electronic  data  collection  systems  permits  collect  ion 
of  data  on  a  near  continuous  basis,  thus  providing  tnu  informa¬ 
tion  required  for  the  sensitive  measures. 

Sensitive  system  performance  measures  are  Owia.nc- 
by  utilizing  a  function  of  a  flight  error  ana  its  rate  of  orange.  Tn.s 
fanction  embodies  the  measurement  principle  “A  sensitive  measure 
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of  an  event  or  condition  is  the  distance  to  that  event  or  condition.” 

This  means,  for  instance,  that  the  conditio'-  ’’error  is  within 
tolerance”  is  replaced  by  the  function  of  the  distance  and  velocity 
to  the  tolerance  condition.  Performance  measures  developed 
from  thu:>  concept  are  able  to  detect  instantaneous  changes  in 
response  patterns.  And,  v.*th  suitable  weighting  functions  arc 
able  to  determine  the  effect  of  that  change  m  the  response  pattern 
on  the  total  mission  performance.  These  measures  arc  J  rmed 
•'system  performance  measures”  since  they  r^.lect  the  effect 
of  moment~to-moment  control  responses  on  the  total  mission 
performance . 

Technical  Objective 

The  technical  objectives  of  the  program,  \r, 
terms  of  the  capabilities  of  the  analysis  model,  are  given  in 
the  following  paragraphs. 

The  purpose  of  this  research  is  the  theoretical 
investigation  of  analytic  methods  for  deriving  differential  weighting 
functions  from  preselected  samples  of  multivariate,  time  sequenceo 
observatior.5  of  aircrew  performance.  The  research  effort  resulted  kn 
an  analytic  model  which  could  be  used  to  prepare  and  to  further 
investigate  differential  weighting  functions  as  a  means  of  establishing 
relationships  between  time  sequenced  observations  of  aircrew 
performance  and  independent  assessments  of  aircrew  proficiency. 

Inputs  to  the  analytic  model  are  multivariate ,  time 
sequenced  data  representing  objective  observations  of  a  flight 
sortie  on  a  designated  NOE  mission  profile.  The  modes  *  an 
empirically-based  processor  where  the  input  data  snot  la  oe  pre¬ 
selected  to  represent  aircrew  performance  across  a  -ange  of 
known  proficiency  levels,  in  most  cases  these  input  caw 
will  have  been  generated  by  some  automated  process,  however, 
this  does  not  exclude  the  possibility  of  data  ootameo  special ueo 
instructor  pilot  observations  or  of  data  selected  to  represent 
other  specialized  features  of  a  given  mission. 

The  model  output  is  a  set  of  weig.  ar.g  functions. 

These  weighting  functions  are  derived  mom  ana*yuu  or  a, renew 
performance  errors  and  rates  of  change  of  error  states  a;*  a 
function  of  designated  locations,  times,  or  utner  operational 
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segments  within  the  relevant  mission  profile.  The  model  can 
provide  an  estimated  distribution  of  aircrew  proficiencies  across 
the  various  functional  segments  wmch  constitute  the  entire  mis¬ 
sion  profile.  Further,  tne  model  can  transform  established 
weighting  functions  into  values  which  are  compatible  with  the 
routine  analyses  of  time  sequenced  data  using  existing  statistical 
models  such  as  linear  regression  or  Fourier  analysis. 

The  derived  weighting  functions  permit  the  fallow¬ 
ing  determinations: 

1 .  Differentiation  of  relative  difficulties  of 
aircrew  performance  across  Known  opera¬ 
tional  segments,  or  time  oeperdent  events 
representing  significant  changes  in  the 
mission  flight  profile  or  mission  require¬ 
ments  . 

2.  Differentiation  of  operational  segments, 
or  time  dependent  events  wmen  r>est  dis¬ 
criminate  among  Known  levels  of  proficiency 
in  given  aircrew  performances  of  a  specified 
mission. 

3.  Establishment  of  design  requirements  for 
subsequent  automation  of  particular  measure¬ 
ment  system  realisations  to  support  routine 
evaluations  of  aircrew  training  performances . 


ANALYSIS  MODE.. 

Met  nod  of  Approver 


Performance  of  manneo  systems  is  lim.teo 
by  our  ability  to  measure  system  ana  component  subsystem 
performance  in  a  reliable  ana  sensitive  manner.  Without 
adequate  performance  measures*  tne  re  ii  no  way  to  produce 
end  test  system  designs,  ,4an  and  execute  training  systems* 
or  evaluate  operational  systems.  Methods  of  developing 
these  performance  measures  can  oe  characterised  by  the  way 
in  which  performance  criteria  are  obtained. 
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One  approach  which  can  be  used  when  all  factors 
that  limit  performance  are  known  and  quantified  is  an  ana¬ 
lytical  method.  For  example,  if  a  problem  requires  that 
an  aircraft  climb  to  a  specified  altitude  while  conserving 
fijel  during  the  climb,  the  criterion,  i.e.,  minimization  of 
fuel,  could  be  precisely  defined  analytically.  Frequently,  how¬ 
ever,  problems  cannot  oe  solved  analytically,  but  demonstra¬ 
tions  of  superior  as  well  as  less  than  superior  performances 
are  available.  In  these  cases  an  empirical  approach  can 
be  used.  This  report  describes  an  empirical  method  for 
analyzing  simulator  flight  data  to  develop  weightings  that  permit 
performance  discrimination  between  two  groups  of  student 
pilots  (one  group  of  students  successfully  passed  the  initial 
Army  rotary  wing  training  program  at  Ft.  Rucker,  Alabama. 
The  other  group  of  students  did  not  pass  that  training  course.). 

Background 

Work  on  the  System  Performance  Measure  Con¬ 
cept  was  initiated  in  the  latter  1960fs  and  first  reported  in 
a  paper  entitled  "A  Theory  of  Adaptive  Man-Machine  Systems 
Applied  to  Automated  Training”  (Connelly,  Schuler,  1969). 

That  paper  presented  the  general  concept  and  theory  of  system 
performance  measurement  within  the  context  of  continuous 
performance  measurement  as  applied  to  a  training  problem. 

At  that  time  only  solutions  to  simple  control  tasks  could  be 
obtained. 


While  development  of  techniques  for  continuous 
measures  has  continued,  the  major  step  in  the  realization  of 
system  performs  nee  measurement  was  the  recognition  that 
the  measures  might  be  derived  empirically  by  the  analysis 
of  performarces  demonstrating  various  levels  of  smII.  The 
work  on  an  empirical  way  of  generating  measures  was  sup¬ 
ported  by  the  U.S.  Air  Force  Human  Resources  Laboratory 
(HRL/ASD).  The  first  report  of  this  work  is  Connelly, 
Schuler,  &  Knoop  (1969). 

The  first  application  study  for  this  empirically 
based  methodology  was  die  development  of  measures  for 


contact  maneuvers  flown  by  undergraduate  student  pilots. 

The  technique  was  at  that  time  referred  to  as  " adaptive 
math  models”  and  was  reported  in  Connelly,  Schuler,  Bourne, 

&  Knoop  (1971). 

Another  application  required  additional  technique 
development  (Connelly,  Bourne,  Loental,  &  Knoop,  1974), 
and  produced  a  computer  processor  which  worked  serm-auto- 
matically  with  the  user  to  generate  performance  measures 
from  performance  data.  This  processor  was  delivered  to 
the  Human  Resources  Laboratory  (ASM)  at  Wright- Patterson 
Air  Force  Base  and  installed  in  the  Sigma  V  computer  at 
that  facility.  The  processor  can  accept  performance  data 
consisting  of  time  samples  of  important  state  variables 
(such  as  the  variables  describing  the  motion  of  an  aircraft, 
and  the  control  inputs)  which  describe  each  performance 
of  a  task.  In  addition,  the  processor  accepts  a  summary 
evaluation  of  each  task  performance.  The  processor 
then  assists  the  user  in  searching  the  data  to  find  dis¬ 
criminate  functions  -which  can  be  used  to  predict  the  sum¬ 
mary  measure  value.  These  discriminate  functions  are 
the  desired  system  performance  measurement  functions. 

The  empirical  technique  has  been  used  to  de¬ 
velop  performance  measures  for  one-on-one  air  combat 
(Connelly  &  Kuhns,  1974),  F-106  attack  mission  (Connellv 
&  Loental,  1974a  and  1974b),  helicopter  navigation  (Connelly, 
Comeau,  &  Laveson,  1975,  and  Connelly,  Comeau,  Bynum,  & 
Holman,  1979),  and  ship  collision  avoidance  (Connelly  & 

Sloan,  1977).  Other  applications  have  included  modeling 
of  decision-making  tasks  and  development  of  measures 
for  fire  department  -  (Connelly  &  Swartz,  1977).  The  method 
was  summarized  for  a  NATO  audience  by  Taylor  and  Knoop  (1972). 

Recently  the  method  was  applied  to  the  development  of  per¬ 
formance  measures  of  Army  teams  using  computerized  tactical  data 
equipment.  This  effort  is  notable  because  the  application  does  not 
require  knowledge  of  the  performance-limiting  effect  of  the  controlled 
equipment  as  was  the  case  with  previous  applications.  Instead,  it 
is  recognized  that  demonstrations  of  performance  implicitly  include 
the  limitations  of  performance,  and  that  the  measures  can  still  be 
derived  using  the  same  methodology.  Another  advance  in  this  effort 
was  the  development  of  the  measure  for  teams  which  are  based  on 
generic  categorization  of  team  tasks  including  team  interactive  tasks . 
This  work  is  documented  in  Connelly,  Comeau,  &  Steinheiser  (1981). 
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MAP  Processor  Description 


The  MAP  Processor  is  a  computer  processor 
that,  under  user  control,  searches  performance  data  seeking 
functional  relationships  among  system  variables  that  permit 
performance  prediction.  The  processor  is  used  where 
demonstrations  of  task  performance  are  available  along  with 
an  independent  assessment  of  the  value  of  each  performance. 
Performance  data  consists  of  the  value  of  system  variables 
typically  sampled  at  uniform  intervals  of  time  from  the 
start  to  the  end  of  the  task.  System  variables  consist  of 
all  variables  needed  to  describe  the  condition  of  the  system 
(i.e.,  the  condition  of  the  process  controlled  by  the  operator) 
at  each  instant  of  time.  System  variables  are  often  referred 
to  as  state  variables  by  control  engineers  and  mathematicians. 
Examples  of  tasks  that  can  be  analyzed  with  the  MAP 
Processor  are  extensive  and  include:  a  pilot  controlling 
an  aircraft  in  a  landing,  a  computer  operator  controlling 
a  computer,  a  computer  programmer  writing  computer 
code,  and  a  project  manager  controlling  a  project. 

The  independent  performance  assessment  can 
be  either  a  subjective  or  an  objective  assessment.  It 
could,  for  instance,  consist  of  a  subjective  rating  reflecting 
a  preference  for  each  performance  demonstration  -  such 
as  scoring  of  a  boxing  match,  ice  skating,  and  diving  per¬ 
formance  in  competitive  sports.  In  other  cases  the  assess¬ 
ment  can  be  objective,  such  as  distance  off  target,  number 
of  data  entry  errors,  time  to  complete  a  task,  and  cost 
to  complete  a  task.  It  is  used  to  order  (or  cluster) 
performance  demonstrations  according  to  performance. 

The  MAP  Processor  then  forms  and  tests  (by  making  Per¬ 
formance  Predictions)  funct  ons  of  the  system  variables 
that  order  (or  cluster)  the  performance  consistent  with 
that  of  the  independent  assessment.  Functions  formed  and 
tested  by  the  Processor  and  found  to  predict  performance 
represent  the  relationships  among  system  variables  and  per¬ 
formance.  The  functions  can  show  how  superior  performance 
(and  other  performance  categories)  was  achieved. 
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The  MAP  Processor  is  designed  in  two  parts. 

One  part,  a  transformer,  converts  the  various  forms  of 
real  world  variables  into  a  standard  form  for  the  search; 
the  other  part  provides  the  search  mechanism. 

A  block  diagram  of  this  system  is  shown  in  Figure 
1  where  the  first  part  or  interface  portion  of  the  process  is 
denoted  as  "Boolean  Questions."  This  system  uses  Boolean 
questions  (questions  which  have  a  yes  or  no  answer)  to  trans¬ 
form  the  real  world  variables  into  a  set  of  Boolean  variables 
wntch  in  turn,  provide  a  standard  form  of  input  to  the  second 
part  of  the  processor. 

Boolean  questions  fall  into  two  categories.  One 
category  is  a  set  of  questions  used  to  identify  the  state  of 
the  process  under  study.  For  instance,  if  we  are  working  with 
an  aircraft  landing  problem,  we  are  interested  in  the  present 
condition  of  the  aircraft.  Thus,  the  state  variables  for  that 
problem  might  include  glide  path  error,  aircraft  velocity,  pitch 
and  roll  angles,  throttle  position,  and  control  stick  position(i.e. , 
all  the  variables  required  to  uniquely  define  the  present 
condition  of  the  aircraft).  Boolean  questions  associated  with 
those  variables  might  ask:  Is  the  glide  slope  error  greater 
than  X  degrees?  Is  the  velocity  error  greater  than  2  knots? 

Four  knots?  Six  knots?  Is  the  pitch  angle  of  the  aircraft  greater 
than  1  degree?  Two  degrees?  Four  degrees?  And  is  the  heading 
of  the  aircraft  aligned  with  the  runway  heading  plus  or  minus 
1  degree?  Two  degrees?  Three  degrees?  When  dealing  with 
aircraft  maneuvers,  one  might  use  Boolean  questions  to  identify 
not  only  the  type  of  maneuver,  but  also  to  identify  the  present  seg¬ 
ment  of  the  maneuver.  As  suggested  above,  numerous  questions 
can  be  asked  to  identify  the  state  of  the  process  under  study. 

It  should  be  noted  that  using  Boolean  questions  formu¬ 
lated  to  identify  the  state  of  the  aircraft  indeed  reduces  the 
amount  of  information  retained  about  the  process  (;.e. ,  instead 
of  using  state  variables,  the  information  is  now  encoded  in 
Boolean  variables).  The  idea  is  to  retain  only  that  information 
believed  to  be  Important  to  performance;  however,  if  an  error 
is  to  be  made,  it  is  best  to  accept  more  information  than  is 
necessary  rather  than  less. 
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Figure  1 .  MAP  Processor 


Considering  now  the  second  Boolean  category 
referred  to  above,  questions  are  also  used  to  represent  the 
known  or  even  suspected  performance  relationships.  For  in¬ 
stance,  if  we  believe  that  the  frequency  of  control  stick  reversals 
is  an  important  indicator  of  performance,  we  can  introduce  a 
Boolean  question  which  asks  if  the  stick  crossed  over  the  center 
position  during  the  last  increment  of  time.  If  we  believe  that  per¬ 
formance  is  a  function  of  the  size  of  deviation  from  the  reference 
path,  we  can  ask  multiple  Boolean  questions  which  indicate  the 
degree  of  deviation  at  a  particular  time.  Further,  if  we  believe 
that  one  variable  is  a  function  of  another  variable  or  set  of  other 
variables  when  superior  performance  is  achieved,  then  we  can 
include  that  candidate  function  as  a  Boolean  question  (e.g. ,  Is  air¬ 
craft  velocity  a  function  of  distance  to  touchdown  plus  or  minus 
specified  error?  In  this  case  the  Boolean  question  would  read; 

Is  V<  KD  +  E  ?  or  V  >  KD  -  E  ? 

where  V  is  the  velocity,  D  is  the  distance  to  touchdown,  and  E 

is  the  specified  error,  K  is  a  constant. 

Typically,  these  candidate  functional  relationships 
are  obtained  from  subject-matter  experts  by  asking  them  what 
they  think  is  important  to  performance.  Other  sources  include 
handbooks  and  textbooks  on  the  subject.  Often,  however,  the 
(Unctions  can  be  developed  empirically  by  analysis  of  data  of 
performances  rated  excellent.  Thus,  if  all  performances 
rated  excellent  (or  superior)  are  grouped  together  and  a 
reference  (Unction  is  determined  such  that  the  function  value 
at  any  particular  instant  is  the  mean  of  all  of  the  demonstra¬ 
tions  rated  excellent  (ana  another  (Unction  computed  as  the 
standard  deviation  (fi)  of  the  performance  rated  excellent),  then 
Boolean  questions  can  be  formed  to  ask:  Do  the  system 
variables  satisfy  that  reference  function  +  1  's'?  +  2 's?  etc. 

These  Boolean  questions  can  be  applied  to  performance  data 
from  any  group  to  determine  the  similarity  of  that  group  to 
those  rated  excellent. 

Once  the  set  of  Boolean  questions  n&s  oeen  defined, 
they  can  be  applied  to  the  real  world  input  data.  Since  ere 
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real  world  input  data  is  time  varying,  the  answers  to  the 
Boolean  questions  will  also  be  time  varying.  As  a  result, 
the  output  of  the  first  portion  of  the  processor  will  be  a  set 
of  Boolean  time  sequences  (BTS)  such  as  those  shown  in 
Figure  2.  At  any  particular  time,  each  Boolean  variable, 
which  is  the  answer  to  its  corresponding  Boolean  question, 
will  either  have  a  yes  or  no  value  which  can  be  expressed  as 
1  and  0  respectively  in  a  digital  computer.  In  this  way,  the 
first  stage  of  the  process  converts  the  real  world  variables, 
which  can  appear  in  various  forms,  into  a  set  of  Boolean  (or 
binary)  variables,  in  a  standard  form. 

Now  consider  the  search  portion  of  the  processor, 
which  incorporates  three  separate  computational  search 
mechanisms.  Each  of  the  sub-processors  searches  for 
and  evaluates  the  utility  of  a  separate  type  of  performance- 
data  characteristic. 

The  first  of  these  three  computational  techniques  is 
called  the  State  Transfer  Technique.  This  technique  is 
designed  to  examine  the  relevance  (to  performance  evaluation) 
of  overall  trends  in  the  performance  as  evidenced  by  transfers 
from  one  system-state  to  another.  A  state  of  the  system 
refers  to  the  status  of  the  total  system  as  represented  by  the 
performance  data  taken  collectively,  i.e.,  it  is  the  values  of 
each  Boolean  variable  at  a  given  instant  of  time.  The  assump¬ 
tion  of  the  State  Transfer  Technique  is  that  performance 
evaluation  is  partially  or  totally  a  function  of  the  dynamic 
change  in  system-states ■,  i.e.,  it  is  based  not  on  the  present 
state  of  the  system,  but  rather  on  tht?  operator  and  system 
responses  that  occur,  given  the  present  state.  The  computa¬ 
tional  task  is  to  determine  a  suitable  state-representation 
of  the  data,  and  compute  the  significance  of  state  transitions. 

The  second  computation  technique  is  called  the  Relative 
Technique.  This  technique  is  designed  to  examine  the  sig¬ 
nificance  to  performance  evaluation  of  relationships  among 
different  performance  variables  as  represented  by  the  Boolean 
time  Sequences  (BTS*s).  The  assumption  of  the  Relative  Technique 
is  that  performance  evaluation  is  partially  or  totally  a  function 
of  specific  (but  presently  unknown)  relationships  between  or 


Sampling  Interval 


1  2  3  4  5  6  7 

BTS1  1  0  0  0  0  0  1 

bts2  0  1  0  0  0  0  1 

bts3  0  0  10  1  1  1 

BTS .  0  0  1  0  0  0  1 

4 

•  • 

•  • 

•  • 

BTSki  0  10  0  10  0 

Note:  A  "I"  represents  a  yes  answer  to  the 
associated  Boolean  question. 

A  ,,0”  represents  a  no  answer  to  the 
associated  Boolean  question. 


Figure  2.  Boolean  Time  Sequences 


among  certain  performance  variables.  (A  very  simple  example 
of  such  a  relationship  would  be  that  which  exists  between  an  aircraft’s 
altitude  and  the  distance  of  the  aircraft  from  the  runway 
threshold  during  an  ILS  approach.)  The  computational  task 
is  to  detect  such  relationships,  if  they  exist,  and  to  discover 
how,  if  at  all,  they  are  relevant  to  performance  evaluation. 

The  third  computational  technique  is  called  the  Absolute 
Technique.  This  technique  is  designed  to  examine  the  sig¬ 
nificance  to  performance  evaluation  of  relationships  between 
temporal  patterns  of  each  system  variable  and  that  of  some 
(presently  unknown)  reference  variable.  The  assumption,  of 
the  Absolute  Technique,  is  that  performance  evaluation  is 
partially  or  totally  a  function  of  certain  relationships  between 
actual  performance  and  a  fivcd  reference  performance.  The 
computational  task  is  to  find  a  suitable  reference  performance 
variable,  to  establish  a  method  of  comparing  it  with  the  actual 
performance  variable,  and  to  determine  the  significance  of 
the  resulting  comparison. 

The  outputs  of  each  of  the  three  major  computa¬ 
tional  models  provide  a  component  of  the  performance  evalua¬ 
tion  -  a  partial  score.  An  additional  analysis,  a  Regression  Analysis, 
is  used  to  combine  the  outputs  from  the  three  computational 
models  to  form  a  single  score-prediction. 

The  next  section  of  the  report  describes  in 
more  detail  the  state  transfer  computation  using  a  phase 
plane  analysis  to  provide  the  Boolean  questions.  Computer 
programs  required  to  implement  this  computation  and  the 
other  two  computational  techniques  are  also  provided. 


METHOD 

As  stated  previously,  only  one  of  the  MAP  sub¬ 
processors  was  used  in  this  effort  -  that  of  the  state  transi¬ 
tion  computation.  The  Boolean  questions  were  generated 
automatically  from  a  phase  plane  analysis  as  will  be 
described. 
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The  analysis  methodology  used  recognizes  that  often 
during  flight  maneuvers  some  variables  are  to  be  maintained 
at  a  constant  rate  of  change  and  that  other  variables  are  to  be 
maintained  at  a  constant  level.  For  instance,  in  "straight  and 
level”  flight  it  is  desired  to  have  altitude,  airspeed,  and  heading 
maintained  at  a  constant  level.  Also,  during  a  climb  it  is 
desired  to  have  airspeed,  and  heading  maintained  at  a  constant 
level*  and  the  rate  of  climb  adjusted  from  0  to  some  specified 
value  and  maintained  at  that  value  until  the  desired  altitude  is 
reached  and  then  the  rate  of  climb  is  to  be  reduced  to  zero  to 
maintain  the  desired  altitude. 

One  way  to  analyze  the  flight  data  to  study  pilot 
proficiency  is  to  observe  how  closely  these  variables  are 
actually  maintained  at  a  constant  level,  and  also  to  observe 
the  technique  used  to  reduce  errors  once  they  occur.  Flight 
errors  may  occur  for  several  reasons  including  rough  air  and 
previous  pilot  error;  but,  it  is  as  important  to  characterize  error 
recovery  procedures  as  it  is  to  characterize  maintenance  of 
low  errors  when  evaluating  pilot  capability. 

An  analysis  tool  that  is  often  useful  in  evaluating 
pilot  performance  is  a  phase  plane  analysis  which  uses  a  plot 
of  the  error  versus  error  rate  of  the  variables  of  interest. 

For  instance,  when  we  are  observing  a  pilot's  control  of 
altitude,  we  plot  the  altitude  error  (which  might,  for  in¬ 
stance,  be  the  deviation  from  a  reference  of  2,000  feet)  against 
altitude  error  rate.  If  the  pilot  is  to  reduce  the  altitude 
error  to  zero  and  to  maintain  it  at  zero,  he  must,  as  will 
be  shcwn^  reduce  both  altitude  error  and  altitude  error  rate 
to  zero  simultaneously.  Thus,  the  phase  plane  shows  hew 
that  objective  is  accomplished. 

Calculation  of  error  rate  from  flight  data  samples 
recorded  at  equal  intervals  of  time  is  illustrated  in  Figure  3. 
Error  rate  is  defined  as  the  change  in  error  divided  by  the 
change  in  time  over  a  specified  time  interval. 

An  example  of  a  phase  plane  plot  is  shown  in  Figure  4. 
Trajectory  A,  in  that  figure,  is  an  oscillatory  trajectory  starting 
in  quadrant  1  where  both  error  and  error  rate  are  positive. 

If  the  positive  error  rate  is  not  reduced,  the  error  would  just 
simply  tend  to  increase  without  bound.  But  if  the  error  rate  is 
reduced  to  a  negative  value,  the  error  itself  is  decreased  as 


A  E 
AT 


Change  in  error 
Change  In  time 


+ 


Initial 
|  Point 


Trajectory  A 


^  \ 


Error 


Time 


1  \ 

Trajectory  B 


Quadrant  4 


4 

/ 


/  Objective 

/  p»** . 


Error 

Rate 


Initial  Point 


< 

w 


Quadrant  1 
^Trajectory  A 


Error 


c  x  «■ - ^Trajectory  B 

I  / 

V  / 


Quadrant  3 


Quadrant  2 


Figure  4.  Trajectories  in  Phase  Plane 
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the  trajectory  moves  into  quadrant  2  (where  error  is  positive  and 
error  rate  is  negative).  If  the  negative  error  rate  is  not  brought 
to  0  in  that  quadrant,  the  system  then  "overshoots"  and  moves 
Into  quadrant  3  where  both  the  error  and  error  rate  are 
negative.  Here  again  the  error  rate  must  be  reduced  to  avoid 
more  negative  errors.  Given  the  pilot  controls  the  aircraft,  to 
make  the  error  rate  positive  again,  he  is  faced  with  the  problem 
of  trying  to  reduce  that  error  rate  to  0  as  the  system  moves 
into  quadrant  4  where  the  error  is  negative  and  the  error  rate 
is  positive.  This  process  can  continue  with  the  aircraft  oscillating 
about  the  reference  altitude  which  corresponds  on  the  phase  plane 
to  a  circular  or  eliptical  pattern. 

The  second  example  trajectory  shown  in  Figure  4 
starts  at  the  same  point  as  did  Trajectory  A  and  is  identified  by 
Trajectory  B.  The  system  starts  with  a  positive  error  rate  which 
is  reduced  to  a  negative  value  moving  the  system  to  quadrant  3j 
but  then  the  negative  error  rate  is  brought  to  0.  This  is  the 
trajectory  of  a  convergence  system  where  the  inittal  positive 
error  is  smoothly  decreased  to  0  without  overshots. 

Of  course  actual  trajectories  may  not  fit  exactly  in¬ 
to  these  two  simple  categories}  but,  these  illustrations  show 
the  relationship  between  the  more  familiar  time  line  trajectories 
and  the  corresponding  phase  plane  trajectories. 

Computerized  analysis  of  the  phase  plane  can  be 
facilitated  by  converting  the  phase  plane  states  into  cells  such 
as  shMm  in  Figure  5.  Here  the  error  and  error  rata  variables 
are  each  divided  Into  five  categories  which  together  make  up 
26  cells  in  the  phase  plana.  The  condition  of  the  aircraft  at 
any  time  with  respect  to  «v  selected  variable  always  falls  in 
exactly  one  cell.  That  is,  if  we  are  observing  altitude,  the 
altitude  error  and  error  rate  can  be  determined  and  consequently 
the  corresponding  cell  in  the  phase  plane  determined.  The  analysis 
procesds  by  observing  ths  pattern  of  cell  transitions.  Thus,  if  the 
aircraft  Is  in  Cell  g,  the  convergent  trajectory  (B)  might  result  in 
the  cell  sequence  6 ,  14,  12,  18,  13.  On  the  other  hand,  oscillatory 
and  divergent  trajectories  produce  other  cell  sequences. 

The  cells  in  ths  phase  plane  can  be  considered  as 
answers  to  two  sets  of  Boolean  questions.  One  set  of  questions 
determines  the  column  corresponding  to  the  error.  And  the 
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other  set  of  questions  determines  the  row  corresponding  to  the 
error  rate.  The  answers  to  these  sets  of  Boolean  questions 
then  identify  the  cell  representing  the  present  condition  of  the 
aircraft. 


Cell  boundaries,  as  determined  by  the  Boolean  questions, 
must  be  established  so  that  the  majority  of  flight  data  falls  within 
the  internal  cells.  A  method  has  been  devised  to  automatically 
establish  all  boundaries  by  scanning  the  data  to  determine  the 
mean  and  variance  of  both  the  error  and  error  rate.  The 
cell  sizes  then  are  selected  as  a  function  of  the  error  and  error 
rate  variances  so  that  the  cell  sizes  can  be  adjusted  auto¬ 
matically  to  accommodate  the  majority  of  the  flight  data. 

Transition  Analysis  Method 


As  an  illustration  of  the  analysis  method,  as  shown 
in  Figure  5,  a  set  of  Boolean  functions  divides  the  error  and 
the  error  rate  (phase)  plane  into  25  regions  or  cells.  These 
Boolean  functions  automatically  determine  the  present  location 
(cell)  of  the  demonstration  data,  which  facilitates  the  state 
transition  computation. 


Several  matrices  are  used  in  the  transition  analysis 
of  operator  performance.  One  is  a  25  X  25  cell  transition  matrix 
whose  elements  are  the  probabilities  of  transfer  from  cell  i  to 
cell  j  from  sample  to  sample.  This  matrix  is  constructed  by 
counting  the  number  of  times  the  system  is  in  each  cell  and 
makes  each  transition.  If  the  resulting  transition  matrix  (T) 
represents  a  regular  Markov  process  (Kemeny  &  Snell,  1960) 
the  state  of  the  system  after  N  transitions,  starting  from  an 
initial  state  (cell)  distribution  represented  by  TT  ,  is  given  by 
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where  T  is  the  state  transition  matrix. 


As  N  approaches  infinity,  there  is  a  limiting  distribution  given  by 
limit  TTn  =  a  (2) 
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where  a  is  the  ensemble  state  distribution  vector  existing  after 
a  targe  number  of  trials.  The  limiting  distribution  can  be  re¬ 
garded  as  the  steady  state  distribution,  such  that 

aT  =  a  (3) 

A  second  matrix  useful  in  analysis  of  operator  control 
policies  with  transition  matrices  is  defined  by  equation  4.  This 
matrix  is  the  weighted  transition  matrix,  such  that  each  element 
is  given  by: 


D;. 

iJ 


=  T..Ct 
IJ  i 


(4) 


The  matrix,  referred  to  as  the  D  matrix,  is  obtained  by  multiply¬ 
ing  each  row  of  the  transition  matrix  by  the  probability  the 
ensemble  will  be  in  the  corresponding  state.  The  elements 
of  the  D  matrix  correspond  to  the  probabilities  that  a  particular 
transition  (transtate)  is  used  in  a  given  control  effort,  and 
are  used  to  generate  a  performance  measure  according  to: 


PM-S  %  Dtj  TSMtj  (5) 

where  TSM  is  a  transtate  score  (weighting)  matrix  whose 
element  values  transform  the  frequency  of  each  transition 
into  incremental  scores  which  are  summed  according  to 
equation  5  to  provide  the  total  score  (PM). 

The  values  of  TSM  are  determined  by  a  method 
described  in  Connelly  &  Loental  (1974a). 

General  Analysis  Procedure 


We  now  consider  the  sequence  of  analyses  performed  on 
the  data  where  each  separate  analysis  is  accomplished  by  a  computer 
program.  The  listings  for  these  programs  are  given  in  Volume  II 
of  this  report. 

Flight  data  for  a  number  of  student  pilots  was  re¬ 
ceived  from  the  ARI  field  unit  at  Fort  Rucker  on  an  IBM 
compatible  9-track  magnetic  tape.  The  tape  was  mounted  on 
a  tape  deck  at  a  time-share  organization.  Specified  files 
from  the  tape  were  read  into  disc  memory  at  the  time-share 
facility.  The  data  was  then  transmitted  over  telephone  lines 
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to  floppy  disc  storage  at  PMA.  In  order  to  efficiently  trans¬ 
mit  the  data,  it  was  converted  from  integer  to  hexadecimal 
format  at  the  timeshare  facility.  After  transmission,  the  data 
was  converted  back  to  integer  format  from  hexadecimal. 

Next,  data  for  each  selected  student  had  to  be  stripped  from 
composit  files  into  separate  student  files.  The  various  parts 
of  a  file  had  to  be  merged  in  order  to  reassemble  it  for  a 
subject.  With  this  data  organization,  the  data  analysis  could 
proceed. 


Figure  6  describes  this  analysis  procedure  in  dia- 
gr*am  fbrm.  The  program  INTHEX  which  was  resident  at  the 
timeshare  facility  provided  the  integer-hex  conversion  and  program 
HEXINT  resident  on  PMA’s  LSI-11  converted  the  data  back  to 
integer  form.  The  merging  of  files  was  conducted  with  a  program 
called  FMERGE .  This  program  accepts  data  from  part-,  of 
many  files  and  constructs  a  new  file  in  accordance  with  a  student 
number.  With  this  program,  data  concerning  a  student  which 
were  located  on  several  different  files  on  several  different  floppy 
discs  could  be  reconstructed  into  a  single  student  file.  At  this 
point  in  the  analysis,  data  for  each  selected  student  were  available 
on  floppy  discs. 

The  next  step  was  to  plot  selected  variables  as  a 
function  of  time  on  a  "strip  line"  type  plot.  The  program 
"STRPLT"  which  provides  this  plot  accepts  control  inputs  to 
identify  the  desired  variables  to  be  plotted.  Variables  plotted 
were:  altitude,  heading,  airspeed,  roll  angle,  and  pitch  angle 
as  a  function  of  time. 

In  addition,  there  are  several  other  utility  programs 
available  such  as  "DFUST"  that  lists  all  data  on  the  terminal 
display,  and  "SEOIO"  which  was  intended  to  locate  particular 
flight  conditions.  However,  it  was  found  that  the  STRPLT  data 
were  sufficient  and  it  was  not  necessary  to  use  SEGID. 

Plot  data  were  analyzed  to  first  identify  the  beginning 
and  end  of  each  different  type  of  maneuver,  and  then  to  identify 
segments  in  each  type  of  maneuver  where  variables  were  supposed 
to  be  maintained  by  the  student  pilot  at  a  constant  level.  Due 
to  the  volume  of  the  strip  plots,  they  were  not  included  in 
this  report  but  were  delivered  to  the  ARI  field  unit  at  Fort 
Rucker  under  separate  cover. 
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The  next  step,  performed  with  PHAN  3,  was  to 
compute  the  mean  and  variance  of  error  and  error  rates  of 
selected  variables  for  students  performing  in  specific  man¬ 
euvers*  The  variables  selected  were:  altitude,  heading, 
airspeed,  and  rate  of  climb*  These  data  were  used  to  set 
the  cell  dimensions  in  the  subsequent  phase  plane  analysis. 

In  order  to  create  a  count  matrix.  Program  PHAN 
5  was  used*  The  approach  was  to  specify  error  and  error 
rate  scale  values  input  by  the  user,  so  that  the  phase  plane 
cell  sizes,  which  were  set  as  a  function  of  the  scale  values, 
were  fixed.  Also,  the  mean  error  for  each  flight  segment 
was  input  into  the  computer  and  subtracted  from  the  segment 
data  values*  These  segment  mean  values  were  taken  from 
tables  presented  in  Appendix  B  and  entered  into  a  computer 
file  to  be  read  by  Program  PHAN  5.  PHAN  5,  of  course, 
also  read  the  flight  data  file  containing  the  time  line  data 
of  the  selected  flight  variables,  and  counted  the  number  of 
times  each  cell  was  used,  i.e. ,  the  number  of  times  transition 
i-^j  was  used*  The  output  was  a  computer  file  containing  the 
count  data* 

Under  user  control,  Program  PHAN  4  read  one  or 
more  count  matrices  and  produced  a  transition  matrix.  For 
instance,  a  transition  matrix  could  be  developed  to  represent 
performance  of  a  single  student  or  alternatively  to  represent 
all  students  in  a  particular  performance  classification. 

As  preparation  for  program  WTMAT,  the  transition 
matrices  for  the  two  performance  groups  are  appended  along 
with  two  target  group  scores  into  a  master  data  file.  The 
organization  of  the  appendix  data  file  is  shown  in  Figure  7. 

Since  data  for  only  two  groups  were  used  in  this  case  the 
appended  file  contains  two  transition  matrices  followed  by 
the  associated  target  scores.  Target  scores  are  used  by  the 
WTMAT  program  as  the  true  score  for  the  associated  group 
and  it  adjusts  weighting  values  to  attempt  to  predict  the 
target  score. 

Next,  Program  WTMAT  read  the  appended  transition 
matrix  file,  determined  the  weighting  required  for  discriminating 
among  the  performance  categories  specified,  and  output  a 
weight  matrix. 
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Finally,  program  SCORE  read  the  weight  matrix  along 
with  a  single  transition  matrix  and  calculated  a  score  according 
to  equation  5.  If,  for  instance,  a  score  is  required  for  a 
number  of  students,  the  program  could  be  run  for  each  student 
using  the  transition  matrix  representing  his/her  performance. 

Analysis  1 


Purpose 


To  determine  sectors  in  flight  maneuvers  corresponding 
to  each  different  type  of  flight  and  compute  mean  and  standard 
deviations  for  error  and  error  rate  for  each  selected  variable. 

Method 


Strip  plots  of  available  data  were  output  from  the 
computer.  The  output,  as  shown  in  Figure  8,  consisted  of  the 
values  of  altitude,  airspeed,  roll  angle,  heading,  and  pitch 
angle  plotted  at  two  second  intervals.  Scales  for  the  Y  axis 
are  output  prior  to  the  plot  and  are  not  shown  in  the  figure. 
Three  columns  on  the  left  of  the  figure  are  from  left  to  right 
respectively,  a  start/stop  of  maneuver  sector  (such  as  "begin 
climb"  or  "climb  has  ended"),  a  maneuver  type  index  (8  shown 
in  the  figure),  and  a  simulator  clock.  To  the  right  of  the 
figure  are  two  columns  which  are  the  simulator  clock  repeated, 
and  a  line  index  number. 

The  line  tndex  number  is  used  to  identify  the  start/stop 
indices  of  maneuver  segments  • 

Results 


Tables  B1  through  B24,  given  in  Appendix  B,  show  for 
6  students  and 4  flight  variables,  airspeed,  heading,  rate  of  climb, 
altitude,  the  start  and  stop  indices,  and  mean  and  standard 
deviations  for  the  error  and  error  rate.  Only  straight  and 
level  flight  was  considered  in  this  analysis.  The  start  and 
stop  indices  and  means  were  entered  into  six  computer  files 
ceded  with  "S"  in  the  fifth  digit  of  the  file  name  (see  Computa¬ 
tional  Plans  in  Appendix  A), 
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Analysts  2 


Purpose 

To  establish  a  method  for  computing  weighting  matrices 
that  permit  discrimination  between  the  two  categories  of  students* 

Method 

Analyses  were  conducted  on  performance  data  of  six 
students  of  which  three  subsequently  had  passed  (P)  the  initial 
entry  rotary  wing  training  course  and  the  other  three  had  not 
passed  (F)  that  course.  The  numbers  of  the  six  students  are: 


1. 

4935 

2. 

20935 

cp) 

3. 

9935 

<p) 

4. 

20933 

(F) 

5* 

8933 

(F) 

6. 

15933 

(F) 

The  code  used  to  identity  each  file  In  the  computer  and 
also  used  for  shorthand  notations  in  this  report  is  defined  in 
Appendix  A.  As  an  example*  the  code*  "A491S1*H  is  read; 

A  Altitude  Data 

49  Student  Number  4935 

Straight  and  Level  Flight 

S_  File  contains  Section  Start  a 

Stop  Indices  (Along  with  Mean 
Values  for  Error  and  Error  Rates 
for  the  Section) 

J_  The  First  File  Using  the  Five 

Digits  Specified 
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The  data  analysis  is  organized  into  six  computational 
plans ,  one  for  each  combination  of  three  aircraft  variables 
(altitude,  airspeed,  and  heading)  and  two  student  groups 
(Pass,  Fail),  These  plans  are  given  in  Appendix  A,  Figures  A2 
through  A8.  Each  computational  plan  gives  the  flow  of  data 
from  the  raw  flight  data  to  the  score.  On  the  left  side  of  the 
diagrams,  in  caps,  are  the  names  of  programs  used  to  obtain 
the  results  (i.e.,  PHAN  3,  PHAN  4,  etc.). 

While  the  data  for  each  student  is  for  straight  and 
level  flight  and  thus  results  are  comparable  with  regards  to 
establishing  the  computational  method,  the  reader  is  cautioned 
that  the  straight  and  level  flight  segments  are  taken  from 
different  flight  maneuvers  for  different  subjects.  Data  for 
the  subjects  who  passed  were  from  early  training  sessions 
but  for  subjects  who  failed,  data  was  taken  from  later  training 
sessions.  Later  sessions  permitted  additional  training  but  also 
involved  more  complex  maneuvers  of  which  straight  and  level 
was  a  part. 


Results 


Table  1  documents  the  error  and  error  rate  scale 
(standard  deviation)  values  used  to  establish  the  phase  plane  cell 
boundaries.  These  values  are  called  for  by  the  PHANS  program. 

Tables  2,  3,  and  4  provide  the  scores  obtained  when 
performance  data  of  all  six  students  were  used  to  determine 
the  score  weight  matrix  for  each  of  the  three  flight  variables. 
Performance  Measure  A  refers  to  (he  measure  derived  using 
all  six  students'  performance  data.  The  scores  show  that 
discrimination  was  possible  within  the  data  used  to  produce  the 
score  weighting  values  (analysis  associated  with  Performance 
Measure  B  values  is  described  in  Analysis  3). 

Although  the  number  of  students  in  each  classification 
was  inadequate  to  draw  conclusions  for  the  general  population 
of  students,  it  was  possible  to  separate  performance  of  the 
students  based  on  analysis  of  only  straight  and  level  flight  data. 
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Table  1 


Flight 

Variable 

Altitude 

Airspeed 

Heading 


Scale  Values  for 


Altitude,  Airspeed 


Heading 


Error 

Scale 


80  feet 

3  k nr  s 


6  degrees 


Error  Rate 
Scale 


8  feet/sec. 


2  knots/sec. 


3  degrees/sec. 


Table  2 


Student  Scores 

Altitude  Control  Performance  in  Straight  and  Level  Flight 


Student  No. 

Pass/Fail 

Score  A* 

Score 

4935 

P 

66.4 

55.4 

20936 

P 

72.4 

69.0 

993$ 

P 

55.3 

49.3 

20933 

F 

33.05 

36.7 

8933 

F 

25.17 

38.3 

15933 

F 

45.87 

41.4 

•  Score  A  was  obtained  using  Performance  Measure  A 
which  used  the  performance  dat e  from  all  six  students 
to  derive  measure  weights. 

**  Score  B  was  obtained  using  Performance  Measure  B 
which  used  the  performance  data  from  students  4995* 
20935,  8933,  and  15933,  t.e. ,  students  from  each 
P/F  category,  to  derive  measure  weights. 


Table  3 


Student  Scores 

Airspeed  Control  Performance  in  Straight  and  Level  Flight 


jtucter*  No. 

Pass/Fail 

Score  A* 

Score  B 

4935 

P 

52.0 

57.5 

20935 

P 

74.8 

73.5 

9935 

P 

83.6 

59.6 

20933 

F 

22.8 

30.8 

8933 

F 

24.3 

25.2 

15933 

F 

32.1 

32.4 

•Score  A  was  obtained  using  Performance  Measure  A 
which  used  the  performance  data  from  all  six  students 
to  derive  measure  weights. 

••Score  B  was  obtained  using  Performance  Measure  B 
which  used  tSrm  performance  data  from  students  4935, 
20935,  8933,  and  15933,  i.e.,  students  from  each 
P/F  category,  to  derive  measure  weights. 


Table  4 


Student  Scores 

Heading  Control  Performance  in  Straight  and  Level  Flight 

Student  No.  Pass/Fall  Score  A*  Score  B*  * 


4935 

P 

57.7 

60.5 

20935 

P 

53.9 

58.9 

9935 

P 

53.3 

46.7 

20933 

F 

39.0 

52.6 

8933 

F 

37.0 

32.6 

15933 

F 

29.0 

CO 

a 

5 

•Score  A  was  obtained  using  Performance  Measure  A 
which  used  the  performance  data  from  all  six  students 
to  derive  measure  weights. 

**  Score  B  was  obtained  using  Performance  Measure  B 
which  used  the  performance  data  from  students  4935, 
20935,  8933,  and  15933,  i.e. ,  students  from  each 
P/F  category,  to  derive  measure  weights. 


Table  5  Illustrates  the  cell  frequency  usage  for 
altitude  control  of  one  group  of  students.  There  are  six 
such  tables  (2  groups  x  3  control  variables)  which  are  given 
in  Appendix  C.  These  data  are  the  25  most  frequently  used 
cell  transitions  and  are  the  corresponding  elements  of  the 
"D"  matrix.  Recall  from  the  theoretical  presentation  that 
the  "D"  matrix  contains  the  probability  that  each  transition 
occurs  given  that  no  additional  information  about  the  process 
was  available,  i.e.,  that  the  existence  of  a  previous  cell 
was  not  known. 

It  is  interesting  to  note  that  the  frequency  of  use 
of  the  transition  13— ^13  which  reflects  the  frequency  of  a 
small  error  and  small  error  rate  was  not  a  good  discriminant 
of  the  two  student  categories  since  students  from  both  groups 
had  a  high  frequency  of  cell  transition  13— M3.  However, 
the  students  who  passed  the  training  course  tended  not  to 
leave  Cell  13  once  it  was  entered. 

The  probability  transition  13— M3  is  closely  related 
to  "time  in  tolerance"  which  is  frequently  used  as  a  performance 
measure  in  control  systems.  As  suggested  by  the  results 
cited  above,  time  in  tolerance  is  apparently  not  a  good  per¬ 
formance  discrimination  factor. 

Tables  6,  7,  and  8  contain  the  weight  matrices 
derived  for  scoring  altitude,  airspeed,  and  heading  performance 
respectively.  Since  all  matrix  entries  were  set  initially  at 
a  value  of  50.0,  it  was  likely  that  those  entries  which  were 
still  at  50.0  were  not  adjusted  to  provide  score  discrimination 
among  the  two  student  groups.  The  maximum  and  minimum 
values  allowed  were  99.0  and  0.0  respectively.  Scores  with 
these  extreme  values  were  apparently  associated  with  cell 
transitions  predominatly  used  by  one  or  the  other  classification 
of  students.  Intermediate  score  values  (v.e.,  between  99.0 
and  5b. 0,  and  50.0  and  0.0)  correspond  to  cell  transitions  used 

infrequently  or  by  both  student  categories.  Better  insight  as 
to  the  importance  of  these  incremental  score  values  can  be 
obtained  in  the  subsequent  analyses. 


Student  20935 


Student  9935 


Student  4935 


Probability 
Cells  of  Usage 


Probability 
Cells  of  Usage 


Probability 
Cells  of  Usage 


13  13  0.0700 

1  1  0.0687 

2  2  0.0487 

5  5  0.0479 

25  25  0.0448 

24  24  0.0336 

12  12  0.0330 

11  11  0.0324 

6  6  0.0307 

21  21  0.0306 

7  7  0.0298 

10  10  0.0293 

18  18  0.0285 

8  8  0.0280 

19  19  0.0261 

23  23  0.0251 

14  19  0.0214 

16  11  0.0207 

21  16  0.0204 

19  14  0.0196 

24  23  0.0165 

12  7  0.0163 

11  6  0,0162 

6  1  0.0139 

19  24  0.0131 
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13 
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Weight  Matrix  Straight  and  Level  Flight:  Altitude  Control 
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Analysis  3 


Purpose 


To  provide  an  initial  estimate  of  the  ability  of  score 
weights  developed  for  one  set  of  students  (i.e.,  a  training 
set  consisting  of  students  who  passed  and  also  students  who 
failed  subsequent  training  programs)  to  discriminate  per¬ 
formance  of  another  set  of  students  (i.e.,  a  test  set  con¬ 
sisting  of  students  of  both  categories  whose  performance 
data  was  not  included  when  developing  the  score  weights). 

It  must  be  noted  that  with  the  limited  number  of  students1 
performance  data  available,  it  was  not  possible  to  de¬ 
termine  the  statistical  significance  of  the  test  set  per¬ 
formance  evaluations;  but  the  analysis  provided  an  initial 
evaluation  of  the  scoring  system. 

Method 


Two  students  from  each  category,  a  total  of  four 
students,  were  selected  at  random  and  their  performance 
data  used  for  the  training  set.  The  training  set  consisted 
of  Students  4935  and  20935  wlx)  passed  the  training  course, 
and  8933  and  15933  who  failed  the  training  course.  The 
two  remaining  students,  one  in  each  category  (9935  passed, 
20933  failed)  were  scored  as  the  test  set. 

Results 


Tables  2,  3,  and  4  show  the  results  of  the  test  and 
training  set  scores.  The  results  are  marked  "Score  B” 
in  the  tables.  Results  for  altitude  control,  Table  2, 
show  that  proper  student  class  discrimination  was  achieved 
fbr  the  test  students  since  Student  9935  was  scored  higher 
than  any  of  the  students  in  the  faded  category,  and  Student 
20933  scored  lower  than  any  student  in  the  passed  category. 
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Turning  now  to  the  scores  for  airspeed  control 
shown  in  Table  3,  we  find  a  similar  result  to  that  of  altitude 
control:  student  class  discrimination  was  achieved  and  the 
effect  of  using  the  two  students’  (9935,  and  20933)  flight 
data  was  similar.  Scores  for  the  remaining  four  students 
were  very  stable  when  comparing  values  of  Score  A  to 
that  of  Score  B.  This  suggests  that  common  factors 
might  exist  in  the  flight  data  for  each  student  class  since 
using  or  not  using  one  student’s  flight  data  did  not  appear 
to  affect  the  scores  much. 

Finally,  turning  to  heading  control  scores,  shown 
in  Table  4,  we  see  that  discrimination  of  student  class 
was  not  achieved.  There  were  several  possible  explanations 
for  such  a  result: 

1 .  Heading  control  was  in  some  way  different 
from  altitude  and  airspeed  control  in  straight 
and  level  flight  (for  instance,  good  heading 
control  may  be  either  very  easy  or  very 
difficult  in  straight  and  level  flight  (SLF); 
therefore,  heading  control  in  SLF  was  not  a 
good  predictor  of  later  flight  training  per¬ 
formance). 

2.  Heading  control  was  not  in  some  way  different 
from  altitude  and  airspeed  control  for  purposes 
of  predicting  student  capability  to  pass  the 
subsequent  initial  rotary  wing  training.  And 
the  results  obtained  with  altitude  and  airspeed 
control  do  not  generalize  for  a  larger  student 
population. 

The  next  analysis  (4)  was  designed  to  investigate  this 
question. 


Comparison  of  student  scores  for  the  three  variables 
under  control  reveals  inconsistent  ratings  of  individual  students. 
For  instance.  Student  4935  ranked  second  in  altitude  control, 
third  in  airspeed  control,  and  first  m  heading  control.  This 
suggests  that  there  may  be  variations  in  the  criteria  students 
used  in  allocating  their  control  effort  to  each  of  the  three 
channels. 
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Analysis  4 


Purpose 


To  determine  the  application  of  score  weights  to  a 
larger  student  population  by  comparing  the  empirically  derived 
scores  to  those  produced  from  transition  weights  suggested  by 
control  theory. 

Method 


Knowledge  of  control  theory  permits  identification  of  some 
cell  transitions  as  "good”  and  therefore  should  be  weighted  high, 
and  others  as  "poor"  and  should  be  weighted  low.  Figures  9  and  10 
show  all  these  transitions  respectively.  Scores  were  calculated 
using  the  control  theory  based  weighting  matrix  for  each  student 
and  each  control  channel,  i.e.,  altitude,  airspeed,  and  heading. 

Results 


Results  of  Analysis  4  are  shown  in  Table  9.  As  with  the 
empirically  based  score  weights,  discrimination  of  student  pass/ 
fail  classification  was  achieved  with  altitude  and  airspeed  control, 
but  not  with  heading  control.  This  is  evidence  that: 

1 .  Students  could  be  classified  with  regard  to  capability 
to  pass  the  subsequent  training  program  by  observing 
their  altitude  and  airspeed  control  ability  in  the 
PASS  (Marco,  Bull,  Vidmar,  &  Shipley,  1979)  simu¬ 
lation  test  program.  However,  recall  that  data 
analyzed  were  collected  from  different  test  sessions 
and  this  alone  may  account  for  the  differences  found 
here.  Additional  student  data  must  be  analyzed  to 
resolve  this  issue. 

2,  There  was  something  different  about  heading  control 
that  prevented  discrimination  of  student  pass/fail 
classification. 

It  should  also  be  noted  that  the  discrimination  for  altitude 
and  airspeed  control  of  pass/fail  classification  was  not  as  good 
(i.e.,  scores  not  as  separated)  with  the  weights  developed  from 
control  theory  as  with  the  empirically  developed  weights.  This 
suggests  that  the  empirically  developed  weights  contain  weights 
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Figure  9.  Cell  Transitions  Assigned  a  Score  of  99.0 


Table  9 


Scores  Using  Weights  Derived  from  Control  Theory 


Student 

Pass/Fail 

Altitude 

Control 

Airspeed 

Control 

Heading 

Control 

4935 

P 

52.4 

51.4 

53.0 

20935 

P 

51.0 

50.4 

48.8 

9935 

P 

52.9 

49.9 

49.4 

20933 

F 

47.9 

49.7 

47.9 

8933 

F 

49.4 

47.9 

50.8 

15933 

F 

50.2 

49.5 

49.3 
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of  factors  consistent  with  good  theoretical  control  (which 
should  permit  extension  of  the  results  to  a  larger  student 
population)  plus  weights  of  factors  which  may  or  may  not 
apply  only  to  discrimination  of  performance  of  the  six  students 
considered  here.  Thus,  there  may  be  additional  factors  to 
be  weighted  which  would  improve  discrimination  of  student 
classification  in  a  larger  student  population. 


MISSION  MODEL 


Method  of  Approach 


It  must  be  noted  that  data  as  to  how  the  actual  near 
term  Scout  mission  functions,  and  the  operations  and  specifications 
of  the  actual  equipment  was  not  available  at  the  time  of  this 
analysis.  Operations  and  specification  data  were  simply 
"imagined"  by  the  authors  for  the  analysis. 


This  section  documents  an  analysis  of  the  effects  of 
operator  errors  on  the  detection  and  destruction  of  an  enemy 
target.  The  process  involved  is  shown  in  Figure  11.  A 
Scout  Helicopter  in  a  masked  position  uses  its  mast  sight  to 
search  for  an  enemy  target.  The  operator  detects  a  target. 

He  then  reduces  the  field  of  view,  lines  up  the  target  on 
the  sight  reticle  and  pushes  a  button  to  obtain  laser 
ranging  on  the  target.  The  result  of  the  laser  ranging 
is  the  calculation  of  the  target  XV 2  coordinates.  This  infor¬ 
mation  is  communicated  to  another  aircraft  which  is  armed. 
The  armed  aircraft  returns,  via  either  an  automatic  link  or 
voice  link,  the  time  of  flight  (TOF)  data,  which  is  manually 
entered  into  a  terminal  guidance  system  on  the  scout  aircraft. 
When  the  missile  is  launched,  the  countdown  is  triggered 
(presumably  manually  in  the  scout).  Near  the  end  of  the 
missile  time  of  flight,  the  target  is  illuminated  by  a  coded 
laser  system  on  board  the  scout  aircraft  to  assist  with  missile 
convergence  on  the  target. 

The  coordinate  system  selected  is  shown  in  Figure 
12.  The  scout  aircraft  is  assumed  to  be  at  zero  X  and  V 
coordinates,  and  at  elevation  Z  above  the  target.  Further, 
the  target  is  assumed  to  lie  in  the  XV  plane  with  X  coordinate 
passing  directly  through  the  target.  Thus,  the  correct  target 
position  as  always  has  a  zero  V  coordinate  value. 
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Hypothetical  Scout  Mission:  Detect  Target/Terminal  Guidance 


Figure  12.  Coordinate  System 


Error  Analysis 


There  are  four  sources  of  error  represented  in 
this  simulation  analysis.  One  is  the  drift  of  the  navigation 
system  which  is  assumed  to  be  reset  by  a  navigation  fix  at 
intervals  of  T  hours.  Thus,  the  location  error  is  given  by 
a  simple  linear  function  of  the  time-since-the-last-navigation- 
fix  (T)  as  follows: 


K1T 

(6) 

K,T 

(7) 

where  K  are  selected  as  random  numbers  from  a  normal 
distribution  where  M  =  0 -and  is  equal  to  a  constant  (C). 

The  second  type  of  error  is  an  error  in  the  angular 
position  of  the  mast  sight  which  can  involve  both  an  elevation 
error  and  an  azimuth  error.  These  error  systems  are  shown 
in  Figures  13  and  14  respectively.  In  calculating  these  errors 
it  is  assumed  that  due  to  the  laser  ranging,  the  range  is 
calculated  accurately  and  introduces  no  error  into  the  system. 
The  calculation  of  an  elevation  error  involves  the  further 
assumption  that  a  misdirected  missile  will  impact  on  the 
horizontal  plane  containing  the  target.  Thus,  as  shown  in 
Figures  13  and  14,  if  the  depression  angle  <*  is  overestimated, 
the  calculated  X  distance  of  a  target  will  be  shorter  than  its 
true  value.  Conversely,  if  the  «  value  is  underestimated, 
the  calculated  X  value  will  exceed  the  true  value.  The 
equations  for  estimating  the  effect  of  the  elevation  error  on 
X  distance  are  as  follows: 

From  Figure  13  it  is  seen  that 

9  =  or  +  <*£  (8) 
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Figure 


and 


X  =  X  -  X„ 
c  T  E2 


(9) 


for  large  X^  and  X_^_  with  respect  to  A  2 


A  2  sr  X  sin0=X  sin(a+or_) 
c  c  Ey 


(10) 


A2  —  X  sin  n 


but 


sin  (or  +  or^.)  =  sin  (a)  COS  (nr  )  + 


(11) 


COS  (or)  sin  (a^) 


and  by  substitution 

XT  Sin  or  (XT  -  XEZ)  (sin  (")  COS  (ir^)  +  cos  (or)  (1^) 
sin  (a 


or  solving  for  X 


E2 


XEZ  **T 


sin  n 


(sin  or  cos  "E  +  cos  a  sin  *^) 


J  (13) 


but  if  is  small,  then  sin  or  is  small  and 


E2 


-  [1-_1] 


(14) 


COS  Or^ 


The  third  type  of  error  which  is  a  data  entry  error 
is  represented  in  two  steps.  First,  a  probability  is  assigned, 
which  is  the  probability  that  a  data  entry  error  will  occur,  and 


in  any  particular  simulation  trial  if  an  error  occurs  then 
the  amount  of  the  error  is  selected  as  a  random  number  from 
a  uniform  distribution  with  a  mean  of  zero  and  a  specified 
range.  Ranges  for  the  errors  in  the  X  coordinate,  Y  coordinate, 
and  2  coordinate  are  established  independently  and  are  given 
in  subsequent  tables. 

The  entry  in  the  time  of  flight  error  which  is  the 
fourth  type  of  error,  is  treated  much  the  same  as  a  data  entry 
error,  i.e.,  a  probability  is  assigned  which  is  the  probability 
that  an  error  occurs  on  any  particular  simulation  run.  If  it  is 
determined  that  an  error  has  occurred,  the  amount  of  the  error 
is  taken  as  a  random  number  selected  from  a  uniform  dis¬ 
tribution  with  a  mean  of  zero  and  a  specified  range.  The  effect 
of  a  data  entry  error  is  taken  in  a  very  simple  form  as  shown 
in  Figure  15.  In  the  figure  the  effect  of  a  time  of  flight  (TOF) 
entry  error  is  a  multiplicative  factor  D.  D  equals  .9  given 
there  is  no  TOF  error.  D  is  reduced  linearly  to  zero  when 
the  TOF  entry  error  is  equal  to  or  greater  than  5  seconds 
(either  early  or  late). 

The  error  effects  discussed  previously  are  sum¬ 
marized  by  the  following  equations: 

=  K  T  +  R  COS  *_  +  D  +  X_..  (15) 

Y„  =  K  T  +  r  sin  +_  +  D_  <16> 

e  2  E  EY 

where  K^T  is  the  effect  of  navigation  drift  since  last  time 
of  fix,  (R  cos  Oi^))  Is  the  effect  of  an  azimuth 
measurement  error,  X  is  the  effect  of  an 
elevation  measurement  error  and  D^.^,  is  the  data 
entry  error. 

The  Y.  coordinate  has  a  similar  set  of  terms  but  does  not 
include  a  term  for  the  effect  of  elevation  error.  Since  it  is 
assumed  that  the  missile  detonates  upon  impact,  and  that  the 
horizontal  plane  contains  the  target,  no  error  in  the  2  axis 
arises  and  thus  the  2  error  is  ignored. 

The  missile  error  measured  as  a  radius  from  the 
target  is  converted  into  probability  of  kill  (P  •  O)  by  the  function 
shown  in  Figure  16.  The  total  probability  of  kill  (P  )  is  given 


a 


Figure  15.  Effect  of  TOP  Entry  Error 
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Figure  16.  Probability  of  Kill  Due  to 
Target  Position  Calculation 
Error 


P  =  P  *  D. 
K  K 


Simulation 


(17) 


A  simulation  has  been  conducted  using  the 
equations  described  above.  This  simulation  is  a  Monte 
Carlo  simulation  with  10,000  runs  per  trial  to  determine  the 
probability  of  kill.  The  program  used  for  this  simulation, 
written  in  FORTRAN  IV,  is  listed  in  Appendix  D. 

Two  series  of  trials  were  conducted.  One  series 
dealt  with  variations  in  each  one  of  the  variables  affecting  a 
probability  of  kill  but  with  mean  heading  error  set  to  zero. 
The  data  for  the  first  series  are  listed  in  Table  10  where 
Item  #3  is  used  as  a  base-line  data  set.  On  each  successive 
run  one  of  the  variables  of  interest  is  changed.  Note  that 
in  Table  10  four  variables  are  changed:  time  since  last 
navigation  fix  (time  since  true),  the  standard  deviation  of 
the  azimuth  and  elevation  errors,  probability  of  data  entry 
error,  and  probability  of  a  time  of  flight  error.  Data 
plots  showing  the  results  of  the  simulation  are  listed  in 
Figures  17,  18,  19,  and  20. 

The  second  simulation  series  used  the  base-line 
data  for  the  four  variables  mentioned  above  but  with  mean 
heading  error  ranging  from  0  error  to  5°  error  in  1° 
steps.  Figure  21  is  a  plot  of  probability  of  kill  as  a 
function  of  heading  error. 


CONCLUSIONS:  ANALYSIS  MODEL 

The  reader  is  cautioned  that  these  results  (stated 
In  Conclusions  2,  3,  4,  5,  and  6)  are  tentative  because  of 
the  limited  data  set  used  and  because  other  factors  related  to 
differences  in  maneuvers  from  which  the  data  was  taken  may 
fUlly  explain  these  results  in  terms  of  other  factors.  Additional 
student  data  must  be  analyzed  to  permit  an  adequate  statistical  analysis. 
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Table  10 


Near  Term  Scout  Mission  Data 


Series  #1 


Run 


Variable 


Value 


1 

2 

3 

4 

5 


Time  Since  True  0.125 

"  0.25 

"  0.50 

"  0.75 

"  0.875 


6 

7 
3 

8 
9 


2 

<r  for 

ii 

M 

If 


.00003 

.00003 

.003 

.03 

.3 


10 

11 

3 

12 

13 


P  (Data  Entry  Error)  .01 

"  .05 

"  .10 

"  .15 

"  .19 


14 

15 
3 

16 
17 


P  (Time  of  Flight  Error)  .01 

"  .05 

"  .10 

"  .15 

"  .19 


vs 


Figure  18.  Probability  of  Kill  vs.  Standard  Deviation  of  Azimuth 


Figure  19.  Probability  of  Kill  vs.  Probability  of  Data  Entry  Error 


Fl^jce  20.  Probability  of  Kill  vs.  Probability  of  Time  of  Flight  Error 


Figure  21.  Probability  of  Kill  vs.  Mean  Heading  E 


However,  based  on  the  results  of  the  four  analyses  at  the 
present  time,  we  conclude  the  following: 

1 .  The  method  for  combining  performance 
data  for  each  group  of  students  and  the 
method  for  calculating  the  transition 
weighting  factors  using  the  WTMAT 
program  appears  to  be  functioning  as 
desired.  A  set  of  weighting  is  produced 
that  discriminates  the  two  groups  of  students 
based  on  the  limited  data  available. 

2.  Altitude  and  air  speed  control  in  straight 
and  level  flight  are  potential  discriminators 
of  the  success  of  students  in  subsequent 
flight  training  programs. 

3.  Because  of  the  stability  of  the  student 
scores  in  air  speed  control,  when  score 
weights  were  generated  both  with  and 
without  flight  data  from  students  selected 
at  random,  analysis  of  air  speed  control 
may  lead  to  a  reliable  prediction  of 
student  flight  training  performance. 

4.  Heading  control,  at  least  in  straight  and 
level  flight,  does  not  appear  to  be  a 
good  predictor  of  a  student* s  ability  to 
perform  well  in  flight  training  programs. 

5.  The  celt  transitions  associated  with  poor 
(and  also  good)  performance  can  be 
identified  with  the  phase  plane  analysis. 

This  information  might  be  translated  into 
aids  for  students  that  do  get  into  the 
flight  training  program. 

6.  The  "time-in-tolerance"  (i.e.,  within 
specified  error  and  error  rate  tolerance, 
such  as  in  Cell  13)  is  apparently  not  a 
good  performance  discrimination  factor. 

This  result  may  be  surprising  since  time- 
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in-tolerance  is  often  suggested  as  a  score 
factor.  However,  this  result  may  be  a 
general  result  since  it  was  noted  previously 
in  a  similar  analysis  but  in  a  totally  dif¬ 
ferent  flight  environment  (Connelly  & 
Loental,  1974).  Also,  ”time-in-toierance” 
is  cited  by  Poulten  (1974)  as  a  measure  to 
be  avoided  in  manual  control  systems 
because  ”in  tolerance”  condition  can  be 
achieved  in  many  ways  in  addition  to  good 
control  policies. 

7.  Success  in  scoring  students  whose  flight 
data  was  not  included  when  developing  the 
score  weights  (i.e.,  the  training  set), 
and  perhaps  even  more  important,  the 
similarity  of  scores  using  empirically 
developed  weights  with  scores  using 
weights  developed  from  control  theory, 
suggests  that  a  "universal”  (i.e.,  one  that 
extends  to  a  larger  set  of  students)  score 
weighting  system  might  exist. 

8.  Empirically  derived  weights  apparently 
contain  weighting  of  factors  consistent 
with  those  developed  with  control  theory, 
plus  additional  factors  which  may  or  may 
not  extend  to  a  larger  group  of  students. 

RECOMMENDAI  l<>|  js:  ANALYSIS  MODEL 

There  are  many  analyses  that  could  not  de  per¬ 
formed  due  to  time  and  funding  limitations,  but  based  on  the 
results  reported  here,  the  following  minimal  set  up  analyses 
are  recommended: 


1 .  Comparison  of  empirically  derived  score 
weightings  for  altitude,  airspeed,  heading 
control,  and  weightings  developed  from 
the  control  theory  should  be  extended. 
These  comparisons  should  be  made  in 
segments  of  straight  and  level  flight  (as 
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the  student  gains  more  experience),  and 
in  other  flight  maneuvers,  turns,  climbs, 
dives,  acceleration,  deceleration,  and  VOR 
tracking.  Further,  these  analyses  should 
be  conducted  using  additional  student 
flight  data. 

2.  A  search  for  a  universal  score  weighting 
matrix  should  be  extended  based  on  the 
additional  analysis  just  described. 

3.  Particular  attention  should  be  given  to 
weightings  based  on  assymetrical  and  non¬ 
linear  control  characteristics  (i.e.,  re¬ 
duction  of  large  errors  vs.  small  errors 
and  favoring  of  positive  errors  over  negative 
errors,  or  vice  versa).  The  purpose  of 
this  analysis  is  to  determine  whether  or 

not  linear  pilot  control  models  and  linear 
analysis  models  can  be  appropriately 
applied  to  these  flight  performance  measure¬ 
ment  problems. 


CONCLUSIONS:  MISSION  MODEL 

Based  on  the  results  obtained  in  the  analysis  and 
simulation  of  a  mission,  we  came  to  the  following  conclusions: 

1 .  It  is  possible  to  provide  an  analysis  and 
supporting  simulation  to  transform  parameters 
representing  human  operator  skills  (including 
manual,  cognitive,  and  interactive  skills) 
into  their  effects  on  mission  performance. 

It  must  be  recognized,  however,  that 
specific  data  on  how  these  systems  actually 

perform  were  not  available  and  that  tthe 
minimal  results  should  not  be  considered  as 
necessarily  representative  of  any  real 
world  system. 

2.  The  next  logical  step  is  to  determine  the 
actual  human  performance  parameter  values 
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and  range  of  parameter  values  by  obtaining 
experiment  data  using  a  validated  detailed 
analysis  method,  or  obtained  from  appropriate 
literature . 


RECOMMENDATIONS:  MISSION  MODEL 

The  following  recommendations  are  made  relative 
to  the  mission  model: 

1 .  Additional  mission  models  should  be 
constructed  so  that  a  more  complete  set 
of  mission  tasks  could  be  analyzed  and 
simulated. 

2.  The  human  operator  tasks  required  to  per¬ 
form  these  missions  should  be  identified, 
and  data  developed  on  the  parameters  of 
task  performance.  (These  parameters 
should  include  time  to  complete  tasks  as 
well  as  the  measurement  of  the  quality 

of  the  task  given  completion.) 

3.  The  systematic  program  for  the  collection  of 
data  from  all  sources  and  the  development 

of  a  performance  handbook  should  be  established. 
The  handbook  should  be  prepared  and  published 
to  support  design  and  evaluation  of  equipment 
for  human  operators. 
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COMPUTATIONAL  PLANS 


Processing  of  data  for  Student  4935  will  be  used  to 
illustrate  the  analysis  method  shown  in  Figures  A2  through  A9. 
The  coding  system  for  naming  files  is  given  in  Figure  A1 . 
Referring  to  Figure  A2,  File  A491S1  contains  Start/Stop 
points,  error  and  error  rate  means  for  heading  control  in 
straight  and  level  flight.  File  F4935  contains  flight  data  for 
Student  4935.  Student  4935  also  has  flight  data  located  in 
a  second  file,  B4935,  and  the  Start/Stop  points,  error  and 
error  rate  means  for  that  flight  data  are  located  in  A491S2 

Using  Program  PHAN  3,  data  files  A491S1,  and 
F4935  are  read  and  a  cell  frequency  count  file,  H491C1  is 
produced.  In  a  similar  procedure,  data  in  File  A491S2  and 
B4935  are  also  read  by  Program  PHAN  3  which  produces  a 
second  count  file,  H491C2. 

Program  PHAN  5  is  used  to  combine  the  two  count 
files,  H491C1  and  H491C2,  into  a  summary  count  file,  H491C3, 
containing  all  the  flight  data  for  Student  4935. 

Program  PHAN  4  is  used  to  convert  Count  File 
H491C3  into  a  transition  file,  H491T3,  representing  all  transi¬ 
tion  probabilities  for  Student  4935  for  heading  control  in 
straight  and  level  flight.  The  summary  count  file,  H491C3,  is 
combined  with  H201C3  and  H991C3  which  are  count  files  for 
Students  4935,  20935,  and  9935  respectively.  Using  PHAN  5, 
the  three  count  files  are  combined  to  produce  HPCI.  HPC1 
represents  cell  transition  counts  for  the  three  students  who 
passed  the  subsequent  training  course.  PHAN  4  is  used 
again  to  convert  the  count  file  HPCI  into  a  transition  file, 

HPT1  • 


As  shown  in  Figure  A6,  Transition  Files  HPT1  and 
HFT1  were  concatenated  with  the  EDIT  program  to  form  file 
HPFT1.  Also,  target  scores  of  80.  and  20.  are  included  in 
File  HPFT1  for  students  who  passed  or  failed  respectively,  in 
the  subsequent  program. 

Program  WTMAT  reads  HPFTl,the  input  file,  and 
produces  HPFW1 ,  the  output  file.  HPFW1  is  a  score  weight 
matrix.  File  HPFW1  is  read  by  the  SCORE  program  and 
the  transition  files  for  each  student  to  give  him/her  scores 
and  the  probability  of  cell  usage. 
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Example  A491S1 


1st  Diqit 

2nd 

&  3rd  Digits 

A  Altitude 

99 

Student  9935 

H  Heading 

20 

Student  20935 

S  Airspeed 

15 

Student  1 5933 

R  Rate  of  Climb 

89 

Student  8933 

49 

Student  4935 

' 

21 

Student  20933 

4th  Diait 

5th 

Digit 

1  Straight  Level 

S 

Start,  Stop,  Means 

2  Right  Turn 

c 

Count  File 

3  Left  Turn 

T 

Transition  File 

4  Normal  Climb 

5  Normal  Descent 

6  Deacceleration 

7  Acceleration 


6th  Digit 

A  number  to  distinguish 
among  files  with  the  same 
first  five  digits 


A491S1  F4935  A491S2  84935  A201S1  F2Q9Q5  A201S2  A20935  A991S1  F9935  A991S2  G9935 


A-3 


Figure  A3.  Computational  Plan  -  Heading 


A491SI  F4935  A491S2  S493S  A201S1  F20935  A201S2  A20935  A991S1  F9935  A991S2  G9935 


Figure  A4.  Computational  Plan  —  A  ('■•speed 
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Figure  A6.  Computational  Plan  -  Score 


Figure  A7.  Computational  Plan  -  Altitude 


A211S1  F20933  A891S1  F8933  A151S1  FI  5933 


Figure  A8.  Computational  Plan  -  Altitude 
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APPENDIX  B 
START/STOP  INDICES 
FOR 

STRAIGHT  AND  LEVEL  FLIGHT 
FOR 


SIX  STUDENTS 


Start 

Stop 

y  of  Error 

/\ 

s  of  Error 

y  Error  Rate 

s"  Error  Rate 

1 

62 

34.31 

16.77 

.46 

4.23 

124 

187 

48.41 

19.69 

.40 

4.16 

244 

262 

4.47 

10.45 

.06 

5.29 

59 

63 

28.80 

1 .92 

-  1 .25 

.50 

93 

111 

44.05 

11.75 

-  2.22 

9.58 

119 

140 

66.73 

7.64 

.10 

4.52 

148 

169 

41.36 

19.71 

-  2.33 

5.43 

169 

175 

25.00 

4.43 

2.00 

1.67 

227 

239 

9.85 

37,54 

6.17 

16.63 

277 

296 

-  85.75 

15.02 

1.74 

8.92 

314 

332 

-433.89 

61.09 

.94 

25.51 

476 

495 

-912.40 

66.68 

-  10.21 

11.18 

496 

504 

-948.00 

26.96 

8.25 

10.83 

554 

565 

29.17 

10.09 

1.09 

5.38 

598 

608 

97.45 

47.70 

12.40 

8.80 

644 

655 

127.83 

55.19 

-  15.55 

9.08 

693 

704 

285.33 

16.42 

5.09 

10.39 

Table  B2 


Straight/Level  Flight  -  Airspeed 
Student  4935 


Start  Stop  y  of  Error  ^of  Error  y  Error  Rate  s  Error  Rate 


Table  B3 


Straight/Level  Flight  -  Heading 
Student  4935 


Start  Stop  y  of  Error 


's'of  Error 


Error  Rate  'S'  Error  Rate 


1 

62 

2.44 

3.27 

.10 

1.03 

124 

187 

1.87 

5.47 

.00 

•1.45 

224 

262 

.30 

4.25 

.09 

1 .45 

59 

63 

3.31 

3.87 

2.40 

.37 

93 

111 

3.57 

11  .51 

.64 

4.60 

119 

140 

2.04 

6.17 

CM 

• 

1 

1.73 

148 

169 

.16 

6.12 

.35 

1.50 

169 

175 

4.70 

3.09 

-  1  .32 

.47 

227 

239 

-  1.20 

12.14 

.17 

9.82 

277 

296 

-182.80 

14.70 

.64 

4.80 

314 

332 

-107.49 

85.83 

2.22 

81.78 

476 

495 

-181.11 

12.75 

2.81 

5.81 

496 

504 

16.98 

12.30 

4.27 

.90 

554 

565 

3.72 

12.35 

-  2.56 

2.49 

598 

608 

-180.20 

5.06 

.83 

3.08 

644 

655 

6.79 

8.83 

-  2.23 

’  .93 

603 

704 

10.75 

23.93 

-  6.02 

2.98 

Table  B4 


Straight/Level  Flight  -  Rate  of  Climb 


Student  4935 


Start  Stop  y  of  Error 


“s' of  Error 


Error  Rate  ^  Error  Rate 


14.47 
-  10.06 
4.37 

-  34.20 

-  84.42 

-  4.91 
64.29 
71.15 
49.60 
20.84 

-319.25 

227.00 

41.92 

409.45 

-508.75 

190.58 


136.56 

134.56 
164.18 

28.35 

325.02 

159.54 
55.97 

203.76 

304.31 

814.13 

356.55 
351.27 
170.25 
301.33 
302.64 
353.02 


-  1.21 

-  2.25 

-  7.56 

-  10.25 
37.11 

•  43 
12.33 

-  9.00 
24.42 
87.28 

-  23.63 

-  17.50 


76.53 
84.06 
107.41 
2 

253.23 
121.05 
4 

181.22 

285.77 

368.61 

265.76 

316.13 

105.45 

295.70 


334.22 


Table  B5 


Straight/Level  Flight  -  Altitude 
Student  20935 


Start  Stop  y  of  Error  's  of  Error  y  Error  Rate  's  Error  Rate 


/\  , 


1 

9 

-340.56 

84.43 

29.75 

8.22 

62 

73 

109.25 

17.85 

5.55 

7.27 

81 

108 

37.89 

93.84 

- 

6.41 

13.63 

112 

123 

86.42 

17.10 

4.45 

2.50 

162 

172 

-719.95 

52.85 

16.60 

3.63 

177 

195 

-404.84 

107.49 

18.28 

4.38 

206 

217 

-  80.50 

48.60 

- 

11. ie 

8.77 

298 

300 

123.67 

8.08 

- 

8.00 

2.83 

622 

652 

-  14.10 

26.83 

- 

1.77 

4.41 

711 

715 

-572.20 

32.15 

- 

20.50 

2.65 

741 

745 

-968.20 

16.42 

10.25 

3.30 

74 

92 

145.05 

-  34.19 

- 

4.33 

11.34 

132 

152 

-  42.33 

314.92 

- 

52.55 

231.78 

209 

228 

-138.50 

50.74 

- 

8.11 

9.14 

327 

334 

-  27.38 

20.67 

- 

8.00 

5.20 

281 

301 

97.19 

34.15 

2.90 

8.32 

446 

465 

-954.90 

14.83 

.68 

5.99 

474 

481 

-873.75 

26.88 

11.00 

2.71 

537 

547 

-  66.09 

17.61 

- 

5.90 

3.81 

587 

597 

-102.36 

19.27 

5.00 

4.69 

625 

636 

9.50 

7.78 

- 

.82 

4.17 

646 

664 

-  55.16 

4.22 

.00 

3.09 

Table  B6 


Straight/ Level  Flight  -  Airspeed 
Student  20935 


Start 

Stop 

1 

9 

62 

73 

81 

108 

112 

123 

162 

172 

177 

195 

206 

217 

298 

300 

622 

652 

711 

715 

741 

745 

74 

92 

132 

152 

209 

228 

327 

334 

281 

301 

446 

465 

474 

481 

537 

547 

587 

597 

625 

636 

646 

664 

y  of  Error 

.67 

-  31.05 

.54 

-  10.10 

3.55 

3.38 

3.70 

3.40 

-  10.70 

4.92 

.96 

-  10.14 

.69 

.09 

-  3.75 
3.23 

-  2.22 
2.85 

-  1.31 
-174.01 

-  8.55 

-  19.33 


of  terror 

1.57 
4.70 
7.50 
1.30 
.78 
1  .70 
1.88 
1.25 
5.55 
.78 
.54 
10.17 
2.37 
4.28 
1.27 
2.48 
3.87 
.95 
2.82 
4.63 
2.90 
2.60 


y  hrror  Kate 

-  .08 
.87 
.22 
.38 
.00 

-  .17 

.11 

1.20 

-  .32 
.45 

-  .30 
1.67 

.39 

-  .51 

-  .43 

-  .06 
.51 
.34 

-  .78 

-  1.55 

.71 

-  .33 


s  Error  Rate 

.1.18 
1 .48 
2.63 
149 
.63 
.61 
1.19 
.85 
1.24 
.30 
.35 
2.51 
1.68 
1.28 
.83 
1,03 

2.70 
.32 
.64 

3.70 
.45 

1.31 


T  able  B7 


Straight/ Level  Flight  -  Heading 
Student  20935 


Start 

Stop 

y  of  Error 

s'  of  Error 

y  Error  Rate 

^  Error  Rate 

1 

9 

44.90 

17.25 

6.04 

2.04 

62 

73 

-201.17 

25.40 

6.87 

.78 

81 

108 

-117.46 

112.65 

6.65 

68.80 

112 

123 

22.74 

4.45 

- 

1.07 

1.11 

162 

172 

15.40 

2.94 

.70 

1.20 

177 

195 

2.65 

4.12 

- 

.37 

.1.36 

206 

217 

-  18.98 

9.55 

- 

2.39 

1.46 

298 

300 

25.94 

6.31 

- 

6.31 

.03 

622 

652 

-  10.85 

7.74 

- 

.90 

1.98 

711 

715 

-572.20 

32.15 

- 

20.50 

2.65 

741 

745 

24.73 

5.45 

3.41 

1.28 

74 

92 

-144.15 

1 1 .26 

- 

.88 

5.53 

132 

152 

-  .63 

11.70 

.83 

3.18 

209 

228 

-  15.39 

14.54 

1.38 

3.75 

327 

334 

-  8.35 

7.89 

- 

2.90 

1.54 

281 

301 

-  63.00 

150.08 

11.93 

80.51 

446 

465 

-  17.38 

5.13 

.80 

3.24 

474 

481 

13.08 

2.66 

1.00 

.37 

537 

547 

-  11.16 

4.36 

- 

1.42 

.80 

587 

597 

-174.01 

4.63 

- 

1.55 

3.70 

625 

636 

-  8.19 

2.50 

- 

.41 

1.34 

646 

664 

-  7.20 

10.45 

1.06 

2.06 

B-7 


Table  B8 


Straight/Level  Flight  -  Rate  of  Climb 
Student  20935 


Start 

Stop 

y  of  Error 

of  Error 

y  Error  Rate 

^  Error  Rate 

1 

9 

917.11 

302.32 

-  87.38 

153.24 

62 

73 

181.25 

223.91 

-  7.73 

157.78 

81 

108 

-197.39 

441 .02 

25.41 

162.33 

112 

123 

142.00 

95.73 

-  25.82 

76.93 

162 

172 

547.36 

129.09 

-  18.60 

100.87 

177 

195 

581.05 

145.62 

14.89 

82.76 

206 

217 

-313.58 

315.32 

-  19.73 

247.23 

298 

300 

-218.67 

168.77 

-163.50 

102.53 

622 

652 

-  50.65 

140.24 

5.13 

100.70 

711 

715 

-659.60 

87.80 

-  46.50 

48.35 

741 

745 

330.80 

160.56 

-111.00 

108.93 

74 

92 

-144.47 

388,03 

-  47.67 

168.55 

132 

152 

8.33 

218.63 

33.50 

147.09 

209 

228 

-251.00 

299.12 

29.32 

221.41 

327 

334 

-243.63 

169.59 

-  5.86 

197.09 

281 

301 

67.71 

289.37 

4.75 

153.33 

446 

465 

19.80 

219.09 

-  4.63 

244.25 

474 

481 

358.38 

94.01 

36.57 

25.22 

537 

547 

-202.91 

142.51 

28.00 

149.22 

587 

597 

186.91 

133.59 

9.30 

155.20 

625 

636 

-  20.17 

137.65 

-  33.91 

90.48 

646 

664 

-  4.47 

102.60 

8.06 

106.99 

B-8 


Table  B9 


Straight/Level  Flight  -  Altitude 
Student  9935 


i 


Start  Stop 

1  62 

62  124 

186  193 

254  272 

324  342 

250  363 

389  395 

512  530 

9  14 

65  76 

113  124 

196  207 

293  299 

365  382 

390  429 

433  440 

520  559 

644  664 

T74  778 

305  844 


y  of  Error 

20.21 

39.75 

-1000.33 

64.95 

7.32 

6.46 

-  992.14 
113.79 

20.33 

26.83 

43.67 

-  859.83 
-1001.71 

-  24.11 

-  2.35 

-  998.25 
152.38 
101.05 

-  566.40 
-1065.80 


s  of  Error 

18.22 

17.78 
.82 

64.65 
22.22 
50.71 

4.57 

39.00 

2,16 

4,39 

16.65 

72.79 
1.72 

12.92 

30.70 
2.12 

44.87 

44.32 

17.71 
92.85 


brror  Rate 

.44 

.50 

.20 

-  5.17 

2.17 
.59 

1.50 

7.00 

.80 

.73 

2.18 
20.45 

.83 

1.59 

2.05 

.43 

2.15 

4.85 

-  11.25 

4.05 


s  Error  Rate 

-  6.44 

5.62 
.84 
16.12 
6.08 
12.11 
2.88 
5.49 
1.48 
2.76 
7.45 

4.82 
.98 

4.37 

4.96 

1.51 

8.83 

6.11 

.96 

17.75 


Table  BIO 


Straight/Level  Flight  -  Airspeed 


Start 

Stop 

y  of  Error 

's'  of  Error 

y  Error  Rate 

's"  Error  Rate 

1 

62 

.10 

2.11 

.00 

.92 

62 

124 

- 

.08 

2.90 

- 

.04 

2.45 

188 

193 

2.20 

.49 

.24 

.54 

254 

272 

5.24 

7.26 

.73 

1.55 

324 

342 

.32 

3.70 

- 

.13 

2.67 

250 

363 

.89 

5.42 

- 

.06 

1 .68 

389 

395 

1.80 

,49 

.GO 

.54 

512 

530 

- 

5.27 

2.73 

.03 

2.91 

9 

14 

- 

.60 

.00 

.00 

.00 

65 

76 

.15 

2.05 

- 

.49 

2.02 

113 

124 

.50 

4.15 

- 

.50 

2.34 

196 

207 

- 

4.35 

2.35 

- 

.65 

.95 

293 

299 

.69 

.23 

- 

.10 

.24 

365 

382 

- 

4.83 

1.33 

.11 

.49 

390 

429 

- 

5.58 

2.78 

- 

.20 

.61 

433 

440 

2.03 

.31 

- 

.09 

.23 

520 

559 

- 

13.01 

5.80 

.43 

1.04 

644 

664 

- 

13.46 

2.23 

.36 

.98 

744 

778 

- 

6.48 

.27 

- 

.15 

.30 

805 

844 

- 

10.67 

6*57 

.54 

1.42 

Table  Bll 


Straight/Level  Flight  -  Heading 
Student  9935 


Start 

Stop 

y  of  Error 

s  of  Error 

y  Error  Rate 

s  Error  Rate 

1 

62 

3.10 

7.80 

.34 

1.74 

62 

124 

1.63 

13.34 

- 

.20 

3.56 

188 

193 

-  2.94 

2.32 

- 

1.22 

.25 

254 

272 

1.22 

14.02 

- 

2.23 

1.88 

324 

342 

-179.70 

8.36 

.76 

3.60 

250 

363 

-  99.28 

122.22 

- 

.02 

47.22 

389 

395 

.08 

.22 

- 

.02 

.22 

512 

530 

-183.42 

4.58 

- 

.34 

2.34 

9 

14 

6.06 

.75 

.39 

.22 

65 

76 

-  1.37 

6.73 

- 

1 .91 

2.59 

113 

124 

16.67 

4.92 

- 

.61 

.  3.85 

196 

207 

.98 

4.45 

1.08 

.47 

293 

299 

-  1.49 

1.03 

- 

.47 

.09 

365 

382 

-  10.99 

3.70 

- 

.64 

1.16 

390 

429 

-  9.53 

7.37 

- 

.22 

2.30 

433 

440 

-  3.04 

1.70 

- 

.71 

.33 

520 

559 

6.02 

17.21 

1 .46 

2.93 

644 

664 

.95 

3.57 

.47 

1.50 

774 

778 

-566.40 

17.71 

- 

11.25 

.96 

805 

844 

-112.30 

69.05 

- 

3.27 

6.79 

Table  B12 


Straight/Level  Flight  -  Rate  of  Climb 
 Student  9935 


Start 

Stop 

y  of  Error 

^»vof  Error 

y  Error  Rate 

^ Error  Rate 

1 

62 

15.61 

207.72 

3.05 

108; 05 

62 

124 

23.22 

187.21 

- 

5.29 

180.90 

188 

193 

7.00 

19.80 

8.40 

16.09 

254 

272 

-124.16 

537.34 

4.06 

219.28 

324 

342 

43.11 

237.15 

- 

22.78 

206.95 

250 

363 

20.59 

394.70 

- 

4.50 

178.38 

389 

345 

37.11 

89.12 

- 

17,83 

77.86 

512 

530 

233.95 

192.45 

- 

4.78 

228.85 

9 

14 

8.83 

63.37 

6.60 

72.84 

65 

76 

26.83 

4.39 

.73 

2.76 

113 

124 

68.33 

230.05 

19.91 

177.95 

196 

207 

644.58 

150.85 

2.00 

121.67 

293 

299 

-  26.86 

32.15 

- 

12.00 

13.46 

365 

382 

53.83 

135.96 

- 

5.94 

61.29 

390 

429 

58.40 

165.69 

3.51 

86.66 

433 

440 

12.25 

39.75 

- 

1.57 

48.61 

520 

559 

76.03 

288.96 

- 

18.72 

87.03 

644 

664 

147.29 

193.76 

1.75 

81.49 

744 

778 

-361.80 

45.53 

- 

29.50 

63.76 

805 

844 

128.45 

570.05 

29.95 

180.67 

B-12 


Table  B13 


Straight/Level  Flight  -  Altitude 


Student 


Table  B14 


Start  Stop  y  of  Error  ^of  Error  y  Error  Rate  's'  Error  Rate 


B-14 


mSSffSgStXS 


Table  B15 


Stratght/Level  Flight  -  Heading 
Student  20933 


Start  Stop  y  of  Error  's'  of  Error  y  Error  Rate  s'  Error  Rate 


30 

-172.65 

5.78 

.30 

3.64 

56 

-  96.28 

144.87 

9.36 

76.76 

82 

6.39 

2.22 

.28 

•  80 

172 

5.04 

3.32 

.55 

1.29 

222 

-156.05 

11.44 

-  1.17 

2.25 

288 

•  32 

.45 

.17 

.15 

365 

1.59 

5.67 

.48 

1.42 

392 

17.44 

7.97 

-  1.13 

1.81 

459 

9.73 

4.30 

.23 

2.03 

562 

3.16 

3.28 

-  .38 

.87 

651 

9.40 

4.68 

.17 

1,73 

AAA 

-186.65 

11.53 

1.57 

.  1.51 

B-15 


9llS!l!8i*«s 


Table  B16 


Straight/Level  Flight  -  Rate  of  Climb 
Student  20933  


Start  Stop  y  of  Eirror  s  of  L-rror  y  Lrror  Rato  's' terror  Rate 


30 

155.78 

299.18 

- 

35.53 

194.93 

56 

-  79.13 

130.27 

- 

8.82 

66.14 

82 

-  28.68 

168.45 

- 

4.33 

72.12 

172 

60.45 

291.10 

- 

37.74 

194.35 

222 

28.40 

214.49 

- 

18.16 

119.32 

288 

28.63 

23.22 

7.86 

31.71 

365 

46.00 

112.91 

.94 

47.57 

392 

173.47 

254.72 

32.78 

181.51 

459 

28.84 

238.32 

10.83 

145.83 

562 

-  34.58 

85.17 

.39 

50.37 

651 

47.40 

267.83 

- 

30.74 

105.95 

AQfi 

wwO 

-  14.37 

106.90 

- 

9.94 

61.29 

B-16 


Hmisstiiiisa*is 

IllSSSSSSlISSsisss 


Table  B17 


Straight/Level  Flight  -  Altitude 
Student  8933 


Start  Stop  y  of  Error  ^of  Error  y  Error  Rate  ^  Error  Rate 


442.00 

31.85 

13.00 

.58 

496.86 

3.08 

1.33 

•  82 

863.62 

40.17 

-  16.29 

1.70 

969.71 

16.94 

6.50 

5.47 

997.12 

20.00 

.24 

5.74 

959.03 

7.11 

.03 

2.97 

41.84 

15.64 

1.67 

6.45 

47.21 

23.15 

-  3.72 

5.90 

56.58 

6.04 

•39 

2.95 

944.16 

21.41 

2.33 

6.19 

285.55 

50.94 

5.95 

10.87 

855.29 

11.64 

.75 

5.89 

1009.40 

6.38 

.41 

2.95 

10.80 

13.67 

1.16 

5.53 

22.47 

20.88 

-  2.11 

8.59 

•  84 

7.12 

.50 

2.46 

974.11 

10.04 

1.94 

5.15 

363.75 

40.94 

-  2.58 

11.48 

B-17 


Table  B18 


Straight/Level  Flight  -  Airspeed 


Student  8933 


Start  Stop  y  of  Error  'y  of  Error  y  hrror  Rate  'SiError  Kate 


30 

37 

- 

3.83 

.31 

.09 

41 

47 

.34 

.59 

-  .30 

77 

84 

- 

3.00 

.32 

-  .09 

96 

102 

- 

9.34 

4.23 

-  1.80 

104 

137 

- 

15.41 

2.95 

.16 

140 

173 

- 

17.72 

1.93 

•  13 

284 

302 

- 

3.25 

4.21 

.60 

330 

348 

- 

1.29 

4.68 

.67 

381 

399 

- 

4.45 

2.19 

.33 

487 

505 

- 

1.86 

2.19 

-  .10 

647 

566 

- 

2.64 

7.10 

1.20 

618 

638 

- 

2.06 

1.07 

•  12 

694 

728 

- 

24.34 

1.34 

•  00 

823 

842 

- 

1.68 

2.64 

•  19 

878 

896 

2.97 

3.96 

-  .03 

928 

946 

- 

•  19 

.80 

.03 

1013 

1031 

1.17 

1.50 

-  ,30 

1073 

1092 

1.71 

7.80 

l.il 

8 


8  t  8  2  *8  8  8  S  £8  8  8  8  8  ±  8  t 


Table  B19 


Straight/Level  Flight  -  Heading 
Student  8933 


Start 

Stop 

y  of  Error 

30 

37 

-188.99 

41 

47 

-  3.28 

77 

84 

-181.97 

96 

102 

-  4.62 

104 

137 

.06 

140 

173 

2.79 

284 

302 

-  1.84 

330 

348 

-183.24 

381 

399 

3.18 

487 

505 

-  .97 

547 

566 

-187.47 

618 

638 

1.94 

694 

728 

.43 

823 

842 

-  3.32 

878 

896 

-179.92 

988 

946 

6.64 

1013 

1031 

.57 

1073 

1092 

-174.^3 

^  of  Error 

y  Error  Rate 

7.46 

2.64 

2.84 

- 

1.34 

1.73 

- 

.75 

1.61 

- 

.79 

6.31 

.08 

4.88 

.06 

3.64 

.03 

4.08 

.20 

1.56 

- 

.37 

2.75 

- 

.00 

9.36 

- 

1.33 

4.67 

.21 

3.77 

- 

.04 

2.43 

- 

.13 

3.69 

.42 

1.49 

- 

•21 

1.08 

.24 

7.16 

- 

1.08 

'S'Error  Rate 


1.92 

.69 

.73 

.40 

2.64 

1.47 

1.54 

3.25 

.46 

1.04 

3.02 

1.68 

1.21 

1.10 

3.97 

.36 

.73 

.82 


B-19 


Table  B20 

Straight/Level  Right  -  Rate  of  Climb 
Student  8933 


Start 

sta> 

y  of  Error 

's’of  Error 

y  Error  Rate 

's' Error  Rate 

30 

37 

414.38 

16.73 

5.43 

19.74 

41 

47 

37.86 

24.91 

7.50 

24.21 

77 

84 

-517.75 

58.26 

23.14 

58.59 

96 

102 

171.57 

193.68 

22.83 

204.61 

104 

137 

8.18 

184.28 

1.88 

102.02 

140 

173. 

•  65 

94.11 

-  5.94 

57.13 

284 

302 

61.05 

214.97 

-  27.28 

128.37 

330 

348 

-106.32 

205.51 

-  17.39 

203.14 

381 

399 

13.95 

95.26 

-  13.33 

63.40 

487 

505 

56.84 

213.94 

9.00 

141.63 

547 

566 

188.15 

357.45 

-  44.32 

154.70 

618 

638 

-  29.71 

193.25 

-  19.20 

119.34 

694 

728 

16.29 

100.43 

-  8.15 

78.49 

823 

842 

43.30 

176.93 

2.89 

101.71 

879 

896 

-  72.53 

287.34 

.50 

246.48 

988 

946 

16.63 

78.14 

-  8.89 

49.62 

1013 

1031 

60.63 

165.75 

-  .94 

118.96 

1073 

1092 

-  66.05 

380.71 

-  43.16 

143.83 

0-20 


SlillSSliSSIliiSsM. 


Table  B21 


Straight/Level  Flight  -  Altitude 
Student  15933 


Stop  y  of  Error 


*suf  Error  y  Error  Rate  "s^Error  Rate 


10 

-  950*60 

16.9^ 

10.50 

.1:29 

52 

-  446.75 

13.68 

6.43 

6.45 

62 

-  502.14 

4.53 

- 

2.00 

3.63 

97 

-  970.63 

16.03 

1.71 

13.60 

109 

-  994.00 

7.84 

3.00 

2.83 

149 

-  908.58 

20.20 

1.84 

5.46 

215 

-  745.79 

35.76 

- 

4.11 

9.38 

281 

122.75 

39.17 

6.06 

4.74 

315 

187.27 

46.05 

- 

3.93 

5.14 

368 

-  5.70 

16.12 

2.05 

6.11 

394 

80.68 

21.86 

- 

2.67 

4.85 

485 

-  928.40 

67.99 

- 

4.16 

18.01 

541 

-  428.80 

39.29 

- 

6.47 

5.22 

591 

-  923.76 

37.99 

2.30 

10.83 

674 

-1299.00 

12.00 

12.00 

.00 

700 

-1107.00 

10.32 

- 

6.25 

1.71 

777 

7.00 

3.65 

- 

1.00 

J  .87 

804 

25.15 

21.77 

3.26 

4.17 

879 

12.89 

5.76 

- 

.5- 

3.28 

975 

-  916.40 

27.37 

17.00 

1.83 

Table  B22 


Straight/ Level  Flight  -  Airspeed 
Student  1 5933 


Start  Stop  y  of  Error  ^of  Error  y  Error  Rate  ^ Error  Rate 


6 

10 

- 

2.88 

.27 

- 

.15 

.30 

95 

52 

- 

8.85 

1.19 

(0 

CM 

• 

.76 

56 

62 

# 

*  ,77 

.29 

.00 

.38 

90 

97 

\  _ 

8.10 

2.05 

- 

.86 

.68 

102 

109 

- 

.52 

1 .41 

- 

.60 

.35 

117 

149 

- 

15.91 

2.22 

- 

.19 

1.02 

197 

215 

.03 

5.6* 

.77 

1.09 

262 

281 

- 

15.57 

3.07 

.41 

.94 

236 

315 

- 

3.40 

5.72 

.54 

.84 

349 

368 

- 

1.56 

2.98 

- 

.54 

.75 

376 

394 

- 

3.95 

4.18 

.60 

.62 

466 

485 

- 

1.50 

7.64 

.41 

2.06 

522 

541 

- 

3.54 

6.14 

1.04 

.89 

571 

591 

- 

2.17 

4.83 

.15 

1.57 

672 

674 

- 

4.20 

.00 

.00 

.00 

OWD 

700 

- 

7.44 

2.87 

1.80 

,49 

768 

777 

- 

2.52 

.47 

- 

.07 

.36 

785 

804 

- 

3.54 

1.94 

- 

.25 

.67 

861 

879 

- 

1.67 

1.19 

.23 

.59 

971 

975 

2.52 

,78 

- 

.45 

.30 

0-22 


Table  B23 

Straight/ Level  Flight  -  Heading 
Student  1 5933 


Stop  y  of  Error 


's'  of  Error 


y  Error  Rate 


's'  Error  Rate 


Table  B24 


Straight/Level  Flight  -  Rate  of  Climb 
Student  15933 


Start 

Stop 

yof  Error 

s'  of  Error 

y  Error  Rate 

's'  Error  Rate 

6 

10 

325.80 

53.99 

14.75 

CVI 

0) 

• 

5 

45 

52 

235.13 

216.54 

• 

lO 

CO 

1 

210.48 

56 

62 

-  81.86 

147.60 

-  65.83 

106.50 

90 

97 

65.38 

440.30 

136.29 

170.61 

102 

109 

97.50 

99.53 

37.57 

29.30 

117 

149 

59.91 

181.03 

4.44 

157.82 

197 

215 

-123.95 

300.51 

6.39 

184.64 

262 

281 

192.95 

151.42 

-  6.00 

86.10 

286 

315 

-122.50 

177.88 

-  19.48 

130.02 

349 

368 

67.25 

195.57 

17.53 

91 .84 

376 

394 

-  77.47 

156.07 

-  1.44 

100.26 

466 

485 

-  86.70 

598.71 

-  3.63 

245.16 

522 

541 

-192.75 

184. 30 

-  5.26 

192.29 

571 

591 

72.86 

34G.02 

-  6.30 

168.97 

672 

674 

344.00 

75.11 

29.00 

169.71 

696 

700 

-158.20 

118.92 

43.25 

160.79 

768 

777 

-  26.20 

69.17 

22.44 

39.74 

785 

804 

93.65 

139.39 

5.89 

88.49 

861 

879 

-  14.05 

112.17 

9.39 

101 .59 

971 

975 

499.40 

110.64 

19.50 

146.98 

APPENDIX  C 


TABLES  OF  PROBABILITIES  OF  CELL  USAGE 


Table  Cl 


Probability  of  Cell  Usage  -  Altitude 
Students  20935,  9935,  4935 


Student  20935 

Probability 
Cells  of  Usage 

Student  9935 

Probability 
Cells  of  Usage 

Student  4935 

Probability 
Cells  of  Usage 

13 

13 

0.0700 

5 

5 

0.0819 

13 

13 

0.0859 

1 

1 

0.0687 

12 

12 

0.0688 

12 

12 

0.0661 

2 

2 

0.0487 

8 

8 

0.0677 

16 

16 

0.0542 

5 

5 

0.0479 

1 1 

11 

0.0655 

8 

8 

0.0399 

25 

25 

0.0448 

13 

13 

0.0528 

5 

5 

0.0391 

24 

24 

0.0336 

15 

15 

0.0521 

25 

25 

0.0385 

12 

12 

0.0330 

18 

18 

0.0472 

11 

6 

0.0324 

11 

11 

0.0324 

6 

6 

0.0457 

7 

7 

0.0299 

6 

6 

0.0307 

7 

7 

0.0323 

6 

1 

0.0283 

21 

21 

0.0306 

25 

25 

0.0316 

19 

19 

0.0276 

7 

7 

0.0296 

12 

7 

0.0256 

9 

9 

0.0271 

10 

10 

0.0293 

21 

21 

0.0192 

1 

1 

0.0267 

18 

18 

0.0285 

20 

20 

0.0190 

10 

10 

0.0227 

8 

8 

0.0280 

8 

13 

0.0155 

8 

13 

0.0209 

19 

19 

0.0261 

7 

12 

0.0148 

14 

19 

0.0190 

23 

23 

0.0251 

7 

6 

0.0148 

19 

14 

0.0181 

14 

19 

0.0214 

17 

12 

0.0137 

9 

14 

0.0163 

16 

11 

0.0207 

18 

17 

0.0131 

4 

9 

0.0158 

21 

16 

0.0204 

16 

13 

0.0131 

20 

25 

0.0148 

19 

14 

0.0196 

16 

11 

0.0121 

2 

7 

0.0143 

24 

23 

0.0165 

10 

5 

0.0118 

1 

6 

0.0133 

12 

7 

0.0163 

19 

19 

0.0118 

1 

2 

0.0133 

11 

6 

0.0162 

19 

18 

0.0118 

25 

20 

0.0126 

6 

1 

0.0139 

1 1 

6 

0.0116 

15 

15 

0.0120 

19 

24 

0.0131 

1 

1 

0.01  Hi 

15 

5 

0.0120 

C-1 


Table  C2 


Probability  of  Cell  Usage  -  Airspeed 
Students  20935 ,  9935,  4935 


Student  20935 

Probability 
Cells  of  Usage 


12 

12 

0.1606 

15 

15 

0.0711 

13 

13 

0.0607 

6 

6 

0.0427 

21 

21 

0.0398 

11 

11 

0.0341 

10 

10 

0.0316 

20 

20 

0.0297 

12 

7 

0.0294 

7 

12 

0.0288 

11 

6 

0.0218 

6 

11 

0.0214 

10 

15 

0.0207 

15 

20 

0.0203 

11 

12 

0.0171 

17 

17 

0.0157 

17 

16 

0.0157 

17 

12 

0.0157 

16 

16 

0.0153 

16 

11 

0.0153 

16 

6 

0.0153 

1 

6 

0.0135 

13 

18 

0.0132 

6 

1 

0.0125 

8 

13 

0.0124 

Student  9935 

Probability 
Cells  of  Usage 


13 

13 

0.1072 

15 

15 

0.0883 

12 

12 

0.0689 

10 

10 

0.0452 

14 

14 

0.0411 

25 

25 

0.0379 

16 

16 

0.0360 

11 

11 

0.0333 

10 

15 

0.0258 

21 

21 

0.0241 

20 

20 

0.0210 

6 

6 

0.0186 

21 

16 

0.0182 

15 

20 

0.0172 

11 

6 

0.0166 

1 

1 

0.0136 

15 

10 

0.0130 

19 

18 

0.0131 

2 

4 

0.0115 

6 

11 

0.0113 

6 

1 

0.0113 

20 

15 

0.0111 

14 

19 

0.0109 

12 

13 

0.0105 

12 

7 

0.0105 

Student  4935 


Cells 

Probability 
of  Usage 

13 

13 

0.1800 

12 

12 

0.0913 

14 

14 

0.0837 

11 

11 

0.0453 

20 

20 

0.0237 

7 

12 

0.0155 

1 

7 

0.0144 

18 

17 

0.0140 

16 

11 

0.0139 

11 

6 

0.0139 

15 

25 

0.0134 

15 

15 

0.0134 

13 

8 

0.0130 

19 

14 

0.0129 

8 

8 

0.0116 

7 

8 

0.0115 

7 

7 

0.0115 

12 

17 

0.0109 

12 

13 

0.0109 

12 

7 

0.0109 

13 

12 

0.0108 

25 

13 

0.0106 

17 

7 

0.0106 

18 

13 

0.0105 

22 

1 

0.0097 
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Table  C3 


Probability  of  Cell  Usage  -  Heading 
Students  20935  .  9935  ,  4935 


Student  9935 

Probability 
Cells  of  Usage 

13 

13 

0.1091 

15 

15 

0.1072 

12 

12 

0.0502 

1 

1 

0.0391 

11 

11 

0.0369 

14 

14 

0.0359 

5 

5 

0.0354 

16 

16 

0.0350 

6 

6 

0.0301 

25 

25 

0.0255 

10 

15 

0.0255 

15 

20 

0.0246 

20 

20 

0.0225 

17 

17 

0.0210 

10 

10 

0.0178 

7 

7 

0.0131 

17 

16 

0.0126 

8 

8 

0.0122 

11 

6 

0.0121 

6 

1 

0.0119 

16 

11 

0.0115 

18 

18 

0.0105 

18 

17 

0.0106 

9 

10 

0.0104 

20 

25 

0.0100 

Student  4935 

Probability 
Cells  of  Usa 


13  13 
12  12 
21  21 

14  14 

25  25 

16  16 
20  20 

15  15 
6  6 

18  18 
5  5 

18  17 

7  8 

16  11 

17  12 

8  13 

7  7 
25  20 
10  10 
11  21 
21  1 

8  8 

15  20 

14  19 

1  25 


0.1100 

0.0570 

0.0552 

0.0528 

0.0520 

0.0449 

0.0333 

0.0266 

0.0261 

0.0246 

0.0230 

0.0207 

0.0197 

0.0181 

0.0180 

0.0167 

0.0161 

0.0152 

0.0145 

0.0140 

0.0136 

0.0123 

0.0120 

0.0109 

0.0107 
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Table  C4 


Probability  of  Cell  Usage  -  Altitude 
Students  15933,  20933,  8933 


Student  15933 

Probability 
Cells  of  Usage 


13 

13 

0.0799 

16 

16 

0.0737 

15 

15 

0.0649 

8 

8 

0.0612 

11 

11 

0.0595 

12 

12 

0.0540 

7 

7 

0.0490 

17 

17 

0.0430 

5 

5 

0.0362 

25 

25 

0.0357 

20 

20 

0.0286 

10 

10 

0.0254 

18 

18 

0.0235 

6 

6 

0.0218 

16 

11 

0.0218 

10 

15 

0.0207 

7 

12 

0.0193 

11 

16 

0.0177 

15 

20 

0.0174 

20 

15 

0.0170 

5 

10 

0.0170 

8 

13 

0.0163 

12 

17 

0.0152 

12 

7 

0.0152 

17 

12 

0.0143 

Student  20933 

Probab  ility 


Cells 

of  Usage 

15 

15 

0.1085 

11 

11 

0.0931 

12 

12 

0.0779 

13 

13 

0.0643 

18 

18 

0.0522 

10 

10 

0.0435 

14 

14 

0.0378 

3 

3 

0.0378 

6 

6 

0.0300 

16 

16 

0.0277 

16 

11 

0.0277 

18 

13 

0.0257 

13 

8 

0.0214 

7 

7 

0.0199 

1 

1 

0.0199 

19 

18 

0.0189 

17 

17 

0.0189 

14 

19 

0.0186 

3 

8 

0.0186 

8 

14 

0.0186 

8 

8 

0.0186 

8 

3 

0.0186 

20 

20 

0.0169 

10 

15 

0.0161 

5 

5 

0.0142 

Student  8933 

Probability 
Cells  of  Usage 


13 

13 

0.1578 

12 

12 

0.1498 

15 

15 

0.1303 

17 

17 

0.0713 

18 

18 

0.0428 

20 

20 

0.0369 

8 

8 

0.0346 

17 

12 

0.0291 

7 

7 

0.0245 

25 

25 

0.0231 

13 

18 

0.0220 

13 

8 

0.0220 

8 

13 

0.0198 

10 

10 

0.0197 

12 

7 

0.0196 

10 

15 

0.0184 

5 

5 

0.0181 

15 

10 

0.0175 

18 

17 

0.0171 

18 

13 

0.0171 

20 

15 

0.0138 

7 

12 

0.0122 

15 

20 

0.0111 

23 

18 

0.0100 

8 

3 

0.0096 
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Table  C5 


Probability  of  Cell  Usage  -  Airspeed 
Students  15933,  20933,  8933 


Student  15933 

Probability 
Cells  of  Usage 

Student  20933 

Probability 
Cells  of  Usage 

Student  8933 

Probability 
Cells  of  Usage 

15 

15 

0.1710 

15 

15 

0.2250 

15 

15 

0.2341 

13 

13 

0.1709 

14 

14 

0.1667 

13 

13 

0^2014 

10 

10 

0.0516 

13 

13 

0.1305 

14 

14 

0.1002 

12 

12 

0.0511 

10 

10 

0.0269 

10 

15 

0.0448 

14 

14 

0.0473 

8 

14 

0.0269 

15 

10 

0.0322 

10 

15 

0.0389 

8 

8 

0.0269 

20 

20 

0.0295 

15 

10 

0.0238 

15 

20 

0.0219 

10 

10 

0.0261 

11 

6 

0.0238 

13 

8 

0.0186 

15 

20 

0.0234 

15 

20 

0.0195 

9 

15 

0.0183 

12 

12 

0.0228 

20 

20 

0.0194 

14 

19 

0.0160 

20 

15 

0.0212 

6 

6 

0.0188 

14 

15 

0.0160 

9 

9 

0.0196 

7 

13 

0.0162 

14 

13 

0.0160 

8 

9 

0.0175 

7 

8 

0.0162 

14 

9 

0.0160 

9 

10 

0.0148 

7 

7 

0.0162 

10 

15 

0.0159 

13 

14 

0.0141 

9 

10 

0.0159 

7 

8 

0.0156 

8 

8 

0.0131 

16 

11 

0.0159 

20 

15 

0.0146 

13 

8 

0.0117 

8 

9 

0.0152 

15 

10 

0.0137 

7 

7 

0.0096 

6 

7 

0.0141 

19 

19 

0.0121 

12 

13 

.0.0090 

11 

11 

0.0141 

19 

13 

0.0121 

19 

18 

0.0081 

18 

18 

0.0131 

8 

13 

0.0106 

7 

12 

0.0071 

14 

19 

0.0127 

13 

14 

0.0102 

7 

8 

0.0071 

9 

14 

0.0119 

12 

12 

0.0101 

14 

19 

0.0058 

20 

15 

0.0112 

5 

20 

0.0099 

20 

24 

0.0053 

8 

14 

0.0102 

5 

5 

0.0099 

24 

17 

0.0052 

12 

16 

0.0094 

20 

10 

0.0098 

9 

20 

0.0049 

Table  C6 


Probability  of  Cell  Usage  -  Heading 
Students  15933  ,  20933,  8983 


Student  15933 

Student  20933 

Student  6933 

Probability 

Probability 

Probabiltt 

Cells 

of  Usage 

1  Cells 

of  Usage 

Cells 

of  Usage 

5 

5 

0.1987 

25 

25 

0.1466 

13 

13 

0.1743 

13 

13 

0.1042 

12 

12 

0.1266 

12 

12 

0.0409 

1 

1 

0.0643 

13 

13 

0.1013 

14 

14 

0.1348 

21 

21 

0.0465 

11 

11 

0.0696 

15 

15 

0.1191 

12 

12 

0.0447 

14 

14 

0.0658 

8 

8 

0.0271 

25 

25 

0.0434 

21 

21 

0.0543 

14 

19 

0.0228 

14 

14 

15 

15 

0.0518 

9 

14 

0.0200 

6 

6 

0.0292 

20 

20 

0.0368 

18 

13 

0.0177 

11 

11 

0.0206 

10 

10 

0.0324 

19 

19 

0.0177 

11 

6 

0.0206 

15 

10 

0.0218 

13 

12 

0.0177 

16 

11 

16 

11 

0.0205 

20 

20 

0.0166 

15 

15 

11 

7 

0.0160 

20 

15 

0.0166 

4 

5 

10 

15 

0.0159 

8 

9 

0.0161 

15 

20 

6 

6 

0.0145 

18 

18 

0.0142 

20 

20 

7 

12 

0.0115 

13 

18 

0.0132 

10 

8 

7 

7 

0.0115 

8 

14 

0.0110 

1 

2 

14 

13 

0.0107 

8 

13 

0.0110 

2 

3 

11 

16 

0.0086 

25 

20 

0.0108 

20 

15 

11 

6 

0.0086 

19 

13 

0.0106 

5 

10 

13 

18 

0.0082 

14 

13 

0.0105 

10 

19 

13 

12 

0.0082 

12 

7 

0.0095 

3 

4 

20 

15 

0.0081 

2 

8 

0.0094 

11 

16 

1 

25 

0.0077 

10 

10 

0.0082 

8 

13 

0.0100 

7 

13 

0.0077 

7 

8 

0.0072 

8 

8 

0.0100 

7 

8 

0.0077 

15 

25 

0.0072 
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